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ABSTRACT 


Searchers and scientists have been studying neural networks for many vears hoping 
to achieve human-like performance in the fields of speech and pattern recognition and 
classification. In this study, we are first going to make an introduction to the field of 
artificial neural networks, then we are going to describe some of the neural nets used in 
the pattern recognition and classification. A computer simulation program from an al- 
gorithmic approach for each one of these networks will be constructed and used to im- 
plement the operation of the net. Its ability will be demonstrated in differentiating 
between different patterns and even correcting a noisy pattern and recognizing it. The 
Hopfield network, the Hamming network and the Carpenter / Grossberg network will 
be individually utilized in developing an algorithm for pattern recognition and classi- 
fication. 

The maximum-lhikelihood sequence estimation function will be mapped onto a neural 
network structure. The application of this structure computations for data detection in 
digital communications receivers Will be described. A computer simulation program will 
be constructed and used to show that neural networks offer attractive implementation 


alternatives for MLSE. 
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I. INTRODUCTION 


A. WHAT IS A NEURAL NETWORK: 

A neural network is a highly parallel network with many interconnections between 
analog computational elements or nodes. In other words, a neural net is a svsten) com- 
posed of many simple processing elements operating in parallel whose function is deter- 
mined bv network structure, connection strengths and the processing performed at 
computing elements or nodes. These nodes offer one possible solution to the problem 
of obtaining the massive parallelism and computational requirements that are presumed 
to be required for such problems as pattern recognition and classification that we are 
going to discuss in this study. Artificial neural nets are of interest primarily because they 
may be able to emulate the speed and performance of real biological neural nets using 
many siniple slow computational elements operating in parallel to obtain high compu- 


tation rates. Figure | illustrates this definition, 


NEURAL NETWORK PROCESSING ELEMENT 





Figure 1. Neural Network and a Nodal Preprocessing Element [Ref. 1] 


A small interconnected neural network is presented on the left side of this figure and 
one simple tvpe of processing element or node is presented on the right side (Ref. 1]. This 


particular node forms the sum of N weighted inputs presented on N input links and 


passes the result through a nonlinearity out on one output link. In addition, the weights 
on the input links can be adapted based on information concerning the correctness of 
the output. Neural nets almost always include an inherent nonlinearity and require 
primarily local connectivity between nodes which are almost always nonlinear, typically 
analog, and mav be slow compared to modern digital circuitry. Nodes may also include 
temporal integration and other types of time dependencies and also mathematical oper- 
ations more complex than summation. [Ref. 1] 

Architectures and processing elements used in neural network models are simplified 
versions of those observed in biological nervous svstems. Figure 2 illustrates a number 


of different tvpes of biological neurons and a small biological neural network. [Ref. 1] 
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Figure 2. Biological Neurons and a Small Biological Neural Network [Ref. 1] 


Characteristics of biological neural networks that artificial neural network models 
hope to provide include: [Ref. 1] 
e Fault tolerance to loss of a small number of computational elements. 
e Insensitivity to small variations between computational elements. 
e The need for primarily local connectivity and local learning rules. 
e Real time response. 


e Parallelism. 


Work in neural networks is generally oriented towards achieving rather high-level 
intelligent functions. such as pattern recognition, categorization, and associative mem- 
ory. The biological knowledge of these functions is far from complete, but it is verv clear 
that neurons and synapses are the fundamental devices used. It is also clear that these 
devices are not programmed in the conventional manner; rather, problem-specific 
knowledge is acquired by a learning process which alters the neuronal parameters di- 
rectly. These are the two principal facts of biology that have been applied to neural 
networks. They are the equivalent of the transistor and of the logically structured pro- 
gram in conventional computers. In addition, the algorithms for calculating the output 
ofa model neuron from its input and the high synaptic connectivity used in model net- 
works both derive from biological observations. [Ref. 1] 

Modern neuroscience provides a great wealth of additional information that has 
only just begun to be applied to neural network modeling. This is because the path from 
this more recent biological information to the desired intelligent functions is relatively 
tenuous, and the simple ideas of neurons, synapses and learning, are themselves sur- 
prisingly powerful. [Ref. 1] 

The few principles of neurons, synapses, and learning constitute the biological 
foundation of most neural networks. They are, of course, insufficient to specify a net- 
work with the kinds of high-level intelligent functions mentioned above. In order to 
achieve these functions, the biological foundation is supplemented with cleverly invented 
ideas, some drawn from other disciplines, notablv physics. This non-biological approach 
is appropriate considering the technological goals of the research, the lack of clear al- 
ternative biological solutions, and the possibility that future research will verify that such 
imported ideas are in fact biological. However, if biological realism is not sufficiently 
maintained, neural networks will lose the ability to interact profitably with neuroscience. 
Heel. 1] 


B. NEURAL NETWORKS IN PATTERN RECOGNITION AND 
CLASSIFICATION : 

Pattern recognition and classification is an area where neural nets have proven to 
be very successful. 

A taxonomy of six important neural nets that can be used for recognition and clas- 


sification of unknown patterns is presented in Figure 3. 


NEURAL NET CLASSIFIERS FOR FIXED PATTERNS 


BINARY INPUT CONTINUDUS-VALUED INPUT 
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| | 
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CLASSIFIER CLUSTERING CLASSIFIER NEIGHBOR CLUSTERING 
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Figure 3. A taxonomy of Neural Network Classification and Clustering Models 
[Ref. 2 


This taxonomy 1s first divided between networks with binarv and continuous valued 
inputs. Below this, nets are divided between those trained with or without supervision. 
Overall, adaptive neural networks can be trained using three types of training proce- 
duress) Rel. i 


e Supervised training, which requires labeled training data and an external teacher. 
The teacher knows the desired correct response and provides a feedback error sig- 
nal after each trial. This is sometimes called reinforcement learning, or learning with 
a critic When the teacher onlv indicates whether a response was correct or incorrect 
and does not provide detailed error information. 


e Unsupervised training, sometimes called se/f-organization , uses unlabeled training 
data and requires no external teacher. Data is presented and internal categories or 
clusters are formed which compress the amount of input data that must be proc- 
essed at higher levels without losing important information. Clustering is an im- 
portant component of manv pattern classification procedures. It 1s sometimes 
called vecror quantization when used to convert analog inputs into a binary form 
suitable for transmission or storage. 


e Self-supervised training is used by automata which monitor performance internally 
and require no external teacher. For example. automata which learn to track a 
moving spot by controlling simulated eve muscles can generate an error signal 


based on the distance between the position of the spot on a simulated retina and 
the center of fovea of the retina. Self-supervision is sometimes called learning-hy- 
doing or learning by experimentation. 

Nets trained with supervision such as the Hopfield net and perceptrons are used as 
associative memories or as classifiers. The teacher provides side information or labels 
that specifv the correct class for new input patterns during training. Most traditional 
statistical classifiers, such as Gaussian classifiers, are trained with supervision using la- 
beled training data [Ref. 2]. Nets trained without supervision, such as the Kohonen’s 
feature-map forming nets [Ref. 2], are used as vector quantizers or to form clusters. The 
teacher does not provide these nets with any information concerning the correct class 
during training. The classical K-means [Ref. 2] and the leader clustering algorithm 
[Ref. 2] are trained without supervision. 

In this study, we are going to focus on the use of neural net classifiers for fixed 
patterns with binary input elements. We are going to implement and simulate, for dif- 
ferent cases of input patterns. the supervised Hopfield net, the Hamming net which 1s a 
neural net implementation of the optimum classifier for binary inputs, and the unsuper- 


vised leader clustering algorithm of the Carpenter / Grossberg net. 


C. NEURAL NETWORKS AS MLSE RECEIVERS OF BINARY SIGNALS IN 
GAUSSIAN NOISE : 

In this studv, we are also going to focus on the use of neural network based 
maximum-likelihood sequence estimation (MLSE) receiver structure. In particular, the 
problem of detecting digital data symbols transmitted over a time-dispersive time- 
varving channel in the presence of additive Gaussian noise will be considered . We are 
going to computer iniplement this neural network structure and simulate it for stationarv 
Or time-varying transmission channel. Results of these simulations will be provided to 


show that neural networks offer attractive implementation alternatives for MLSE 
ef. 3}. 
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Il. THE HOPFIELD NETWORK 


A. GENERAEMTEs: 

In recent years, an upsurging interest in neural networks made of highly parallel 
computational elements connected in patterns that are reminiscent of biological neural 
nets has caught attention of researchers and scientists. In particular, more recent work 
has explored the ability of a neural model described by Hopfield to serve as a content- 
addressable memory (classifier). This network retrieves one of the M stored exemplars 
given an input pattern which is a noisy version of one of these exemplars. A classifier 
determines which of the M exemplar patterns is most similar to the noisy input pattern. 

In the following study, we will focus on the classification problem because a 
content-addressable memory 1s essentially a classifier which outputs the exemplar for the 
selected class instead of an index to the class. Classification is a fundamental operation 
that is essential to the important problem of speech and image recognition whether 
achieved by biological or artificial means. 

Past studies have demonstrated that the Hopfield model can be used as a content- 
addressable memorv for random input patterns and to classify binary patterns created 
from radar cross sections, consonants and vowels extracted from spoken words, and lines 
in an image. These results demonstrate that a neural network based on the Hopfield 
model can perform classification. In addition, Hopfield models have been successfully 
applied to other problems, such as, the traveling salesman problem, the Analog to Dig- 


ital (A-D) converter problem, and the signal decomposition problem. [Ref. 4] 


B. OPERATION OF THE HOPFIELD NETWORK: 

The unit used by a number of scientists 1s the familiar binarv threshold unit (Mc 
Culloch-Pitts neuron) whose output is | if and only if Fis = 0, where 0s) Si — 
otherwise 0, where N is the number of elements or bits ina pattern, s,is the current value 
of the ith input and w, is the corresponding synaptic weight from 1 to unit } whose 
threshold is 6, In the McCulloch-Pitts networks, every neuron processes its inputs to 
determine a new output at each time step [Ref. 5]. By contrast, a Hopfield net is a net- 
work of such units subject to the updating rule: “Pick a unit at random. If the sum of 
the weights on connections to other active units is positive, turn it on. Otherwise turn 


it off’. The operation of this network is described as we first apply input values of an 


unknown pattern at time zero through the bottom threshold-logic nodes. This forces the 


output of the net to match the unknown pattern at time Zero: 
uO) = x; O<i<N—-]1 (2-1) 


where u(t) is the output of node 1 at time t and x, is element 1 of the input pattern taking 
on the values +1 or —1. Following this initialization, the network iterates in discrete 


time steps using the given equation: 


N=} 


BAC + 1) =f D0 1 | O<jsM-1 (2 — 2) 


i 
In this equation f, is a modified hard-limiter function and w, is the weight applied 
to the output of node 1 that feeds to node j. Previousely, we have assumed the elements 


of the input vector x take on values +1 and —1, respectively, for the +1 and —1 states, 


then f, 1s the symmetric hard-limiting function. 


a +1 ,ifa>0 
fie ‘fade (2-3) 





Figure 4. The hard-limiting function used in the Hopfield network 


The weights are set using exemplar patterns for all stored classes. 


M-1 
> ax} ix] O-4) 


0 i=j O<ij<N-1 


Where x is element 1 of the exemplar for pattern s. The output of each node is fed to 
everv other node with a weight that 1s symmetric, and each node does not feedback to 
itself. After convergence, the output of the net is the final pattern represented by the 
outputs of the nodes. 


x', = p(co) FeO = I 2 = 


The network 1s considered to have converged when the outputs no longer change 
on successive iterations. When the Hopfield net is used as an associative memory, the 
network output after convergence is used directly as the restored memory. When used 
as a classifier, the output of the Hopfield net after convergence must be compared to the 
M exemplars to determine if it matches an exemplar exactly. If it does, the output is the 
exemplar that best matches the output pattern. If it does not, then a “no match” result 
OCCUnS. 

Hopfield first demonstrated that when the net is trained with M exemplar patterns 
using Equation 2-4, and an exemplar is presented at time zero, then the final pattern in 


the net after convergence will be one of the M exemplars with a high probability if, 
M<O.15N (2 — 6) 


The exemplars thus form stable states of the net. Hopfield’s statistical results were 
obtained with randomly generated exemplars. It 1s possible and relatively easv to select 
a set of M exemplars that satisfies Equation 2-6, but does not form stable states in the 
Hopfield net. These exemplars must have many elements in common. When an exemplar 
for one of these patterns is presented at time zero, the network does not converge to any 
of the trained exemplars. Instead, it converges to a spurious pattern never seen before. 
This problem of spurious states also occurs when a noisy exemplar is presented to the 
net. Even when the M exemplars are stable states of the net, there 1s no guarantee that 
noisy versions of these exemplars passed through discrete, memoryless channels and 
presented at time zero will converge to the original exemplars. Hopfield, for example, 
observed that the number of spurious states found increases substantially as more ele- 


ments in the input exemplar are corrupted. 





The Hopfield neural network can be used as a classifier only when: 


1. The exemplars for the patterns to be classified form stable states and converge to 
themselves when presented at time Zero as input. 


2. A mechanism 1s provided to determine which of the M exemplars the net is closest 
to after convergence. 

The first requirement is a necessary condition for a proper classification operation. 

The second is necessary because the Hopfield net bv itself is not a neural-net classifier, 

but is more like a preprocessor which still requires a classification net to select which of 


the M classes an output pattern is closest to . 
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Figure 5. The Hopfield net used as a content-addressble memory [Ref. 2] 


It is difficult to satisfy the requirement that exemplars form stable states without 
actually running the Hopfield net. In general, patterns that are more random will satisfy 


this requirement more easily than patterns with many bits in common. 


C. IMPLEMENTATION OF THE HOPFIELD NETWORK: 
The Hopfield network can be used either as an associative memory or as a content- 


addressable memory which is described in this study. The Hopfield net shown in Figure 


5, has N nodes containing hard-limiting nonlinearities and binary inputs and outputs 
taking on the values +1 and —1. The output of each node is fed back to all other nodes 
via Weights denoted w;,,. 

A computer algorithm to implement the operation of this net as a content- 


addressable memorv can be summarized in four necessarv steps: [Ref. 2] 


Step 1. Assign Connection weights : using Equation 2-4 


Step 2. Initialize with Unknown Input Pattern: using Equation 2-1 


Step 3. I[terate Until Convergence : using Equation 2-2, the process is repeated until 
node outputs remain unchanged with further iterations. The node outputs then represent 


the exemplar pattern that best matches the unknown input. 


Step 4. Repeat by Going to Step 2 


The weights are first set using Equation 2-4 and elements of the M stored exemplar 
patterns as the operation algorithm of the net stated in the first step. Eight patterns 
(M=8), shown in Figure 6, have been selected to simulate this algorithm and were 
stored in the memory of the network. 

For convenience, these eight patterns were selected to be 120 nodes (12 by 10 ma- 
trices) each. The onlv limitation in the choice of N (number of nodes) is the time that 
the net will take to iterate and converge to an output pattern or respond with a “no 
match”. After assigning connection weights, an unknown input pattern is imposed on the 
net at time Zero by forcing the output of the net to match the unknown pattern. Then, 
the net iterates in discrete time steps. The net is considered to have converged when 
outputs no longer change with further iterations. The pattern specified by the node 
outputs, after convergence is reached, is the net output. [Ref. 4] 

Using the Hopfield net as a classifier, the output will be compared to every one of 
the M class patterns. If the output matches an exemplar, the classification is terminated 
and the output is that class whose exemplar best matches the output pattern. If it does 


not, then a “no match” result occurs. 
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The Eight stored patterns 


Figure 6. 
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D. SIMULATION OF THE HOPFIELD NETWORK: 

Using the Fortran program provided in Appendix A, we simulated the operation of 
the Hopfield net when used as an addressable-content memory (classifier) for different 
inputs. For convenience and simplicity, the M class patterns were first introduced to the 
net as a matrix of N by M (120 by 8), where each vector colunm of 120 nodes represents 
the 12 by 10 matrix representation of the class. The first vector colunm represents the 
pattern of a ‘0’, the second of a ‘1’, the third of a ‘2’, the fourth of a 3, the fifth) of aeam 
the sixth of a ‘6’, the seventh of the block pattern representation of the ‘point’ and the 
eight and last of a ‘9°. The elements of each class pattern take on the values of +1 for 
a’ black pixel “and —1 fora’. ’ 

The behavior of the network is simulated first by presenting the pattern of digit ‘3’ 
aS an input pattern. To make it more interesting, a corrupted version of this pattern is 
achieved bv randomly reversing each bit, of the matrix representation of digit 3, inde- 
pendently from +1 to —1 and vice versa with a probability of 0.25. Implementing this 
pattern 1s equivalent to receiving noise corrupted bits of a digit in a noisy communi- 
cation channel. 

The corrupted input pattern was then imposed on the net at time zero. After the 
first iteration, the net still can not tell which class the input pattern corresponds to. As 
more iteraticns took place, the output becomes more and more like the correct exemplar 
pattern of the digit 3, as you can see in this simulation result provided in Figure 7. 

Then, at the third iteration, the net has converged and the output, as can be seen, 
is indeed the pattern of the digit 3. Onlv three iterations were sufficient for the net to 
converge to the corrected digit and to recognize it as a 3. Now, we tried to see if we 
present a corrupted pattern of another digit, how manv iterations will be used to con- 
verge to the correct result? Will it take only three iterations to do so ? The input pattern 
that we used was of digit 9° and using the same procedure we randomly reversed each 
bit from +1 to —1 and vice versa with the same probability and error distribution as 
before. The response of the network to this input pattern is illustrated in Figures 8 and 
2). 


THE UNKNOWN INPUT PATTERN TO THE HOPFIELD NETWORK, ON THE LEFT AS IM- 
POSED ON THE NETWORK AND ON THE RIGHT IN A MUCH CLEAR REPRESENTATION 
WHERE EVERY ( # ) REPLACES A 1 AND EVERY ( . ) REPLACES A -1: 
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THE OUTPUT OF THE HOPFIELD NETWORK LOOKS LIKE THE FOLLOWING FOR THE 
UNKNOWN INPUT PATTERN PRESENTED. THE PATTERN ON THE LEFT CORRESPONDS 
MeetTHE OUTPUT AFTER THE 1ST ITERATION WHILE THE PATTERN IN THE MIDDLE 
CORRESPONDS TO THE OUTPUT AFTER THE 2ND ITERATION, THE PATTERN ON THE 
RIGHT CORRESPONDS TO THE OUTPUT AFTER THE 3RD ITERATION. 
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CLASSIFICATION OF THE UNKNOWN INPUT PATTERN: 





AT THIS POINT, FURTHER ITERATIONS WON'T MAKE ANY CHANGE ON THE OUTPUT 
OF THE NETWORK AND THE PATTERN SPECIFIED BY THE OUTPUT NODES IS THE 
NET'S OUTPUT. THE TASK OF THE NET NOW IS TO CLASSIFY THE INPUT AS AN 
ALREADY KNOWN PATTERN OR A NO MATCH WILL OCCUR. 

AFTER CLASSIFICATION, THE OUTPUT PATTERN OF THE HOPFIELD NET MATCHES 
BEST THE PATTERN OF DIGIT THREE. 


Figure 7. Hopfielf net response to the first input pattern 
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THE UNKNOWN INPUT PATTERN TO THE HOPFIELD NETWORK, ON THE LEFT AS IM- 
POSED ON THE NETWORK AND ON THE RIGHT IN A MUCH CLEAR REPRESENTATION 
WHERE EVERY ( =# ) REPLACES A 1 AND EVERY ( . ) REPLACES A -1: 
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THE OUTPUT OF THE HOPFIELD NETWORK LOOKS LIKE THE FOLLOWING FOR THE 

UNKNOWN INPUT PATTERN PRESENTED. THE PATTERN ON THE LEFT CORRESPONDS 
TQ THE OUTPUT AFTER THE 1ST ITERATION WHILE THE PATTERN ON THESRIGes 
CORRESPONDS TO THE OUTPUT AFTER THE SZND ITERATOR: 
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THE OUTPUT OF THE HOPFIELD NETWORK AFTER THE 3RD ITERATION IS PRESE= 

NTED ON THE LEFT WHILE THE PATTERN ON THE RIGHT CORRESPONDS TO THE 

OUTPUT AFTER THE 4TH ITERATION. 
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Figure 8. Hopfield net response to the second input pattern 
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CLASSIFICATION OF THE UNKNOWN INPUT PATTERN: 


AT THIS POINT, FURTHER ITERATIONS WON'T MAKE ANY CHANGE ON THE OUTPUT 
OF THE NETWORK AND THE PATTERN SPECIFIED BY THE OUTPUT NODES IS THE 


NET'S OUTPUT. THE TASK OF THE NET NOW IS TO CLASSIFY THE INPUT AS AN 
ALREADY KNOWN PATTERN OR A NO MATCH WILL OCCUR. 

AFTER CLASSIFICATION, THE OUTPUT PATTERN OF THE HOPFIELD NET MATCHES 
BEot tHE PATTERN OF DIGIT NINE. 





Figure 9. Classification response of the Hopfield net to the second input pattern 


Then, using the same probability of error and distribution of errors in the input 
pattern as for digit ‘3’, we conclude from the result of this simulation that the number 
of iterations needed to get the right answer depends on how close the input pattern was 
to a stored one. This may explain why there 1s a difference in the number of iterations 
taken by the net to converge to the nght answer. We may say that the corrupted input 
pattern of digit ‘5’ was closer to the perfect exemplar for the °3’ than the noise disturbed 
input pattern of digit ‘9’ is to the perfect ‘9’ using the same noise corrupted bits distrib- 
ution and the same probability of error as for digit °3’. 

Using the noise disturbed pattern for digit ‘9° as input to the net, but now with dif- 
ferent error distribution and the same probability of error (0.25), the simulation and re- 
sults were as shown in Figure 10. 

The result shows perfectly that the number of iterations that the network needs to 
converge depends on the error distribution in the input pattern and not on the proba- 
bility of error. 

Now as a conclusion after all these simulations and results, one could say that when 
imposing a perfect input pattern on the net, the network will take only one iteration to 
recognize it as a stored one. To verify this finding, we take the pattern of digit ‘2° and 
present it as the input pattern to the net at time Zero without making any change in Its 
elements (a perfect exemplar of digit ‘2’). The response of the net to this perfect input 


pattern 1s presented in Figure 11. 
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THE UNKNOWN INPUT PATTERN TO THE HOPFIELD NETWORK, ON THE LEFT AS IM- 
POSED ON THE NETWORK AND ON THE RIGHT IN A MUCH CLEAR REPRESENTATION 
WHERE EVERY ( # ) REPLACES A 1 AND EVERY ( -. ) REPUACHES eee 
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THE QUPUT OF THE HOPFIELD NETWORK LOOKS LIKE THE FOLLOWING FOR THE 
UNKNOWN INPUT PATTERN PRESENTED. THE PATTERN ON THE LEFT CORRESPONDS 
TO THE OUTPUT AFTER THE 18ST ITERATION WHILE THE PATTERN IN THE MiDiie 
CORRESPONDS TO THE OUTPUT AFTER THE 2ND ITERATION, THE PATTERN ONeaae 
RIGHT CORRESPONDS TO THE OUTPUT AFTER THE 3RD ITERATION. 


CLASSIFICATION OF THE UNKNOWN INPUT PATTERN: 





AT THIS POINT, FURTHER ITERATIONS WON'T MAKE ANY CHANGE ON THE OUTPUT 
OF THE NETWORK AND THE PATTERN SPECIFIED BY THE OUTPUT NODES {3S TEs 
NET'S OUTPUT. THE TASK OF THE NET NOW IS TO CLASSIFY THE INPUT AS AN 
ALREADY KNOWN PATTERN OR A NO MATCH WILL OCCUR. 

AFTER CLASSIFICATION, THE OUTPUT PATTERN OF THE HOPFIELD NET MATCHES 
BEST THE PATIERN@OF SOiGid Nine 


Figure 10. Hopfield net response to the third input pattern 
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THE UNKNOWN INPUT PATTERN TO THE HOPFIELD NETWORK, ON THE LEFT AS IM- 
POSED ON THE NETWORK AND ON THE RIGHT IN A MUCH CLEAR REPRESENTATION 
WEeke BVERY ( #® ) REPLACES A 1 AND EVERY ( . )° REPLACES A -1: 
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THE OUTPUT OF THE HOPFIELD NETWORK LOOKS LIKE THE FOLLOWING FOR THE 
UNKNOWN INPUT PATTERN PRESENTED. THE PATTERN SHOWN HERE IS THE NET'S 
erreur AFTER THE 1ST ITERATION. 


CLASSIFICATION OF THE UNKNOWN INPUT PATTERN: 





AT THIS POINT, FURTHER ITERATIONS WON'T MAKE ANY CHANGE ON THE OUTPUT 
OF THE NETWORK AND THE PATTERN SPECIFIED BY THE OUTPUT NODES IS THE 
NET'S OUTPUT. THE TASK OF THE NET NOW IS TO CLASSIFY THE INPUT AS AN 
ALREADY KNOWN PATTERN OR A NO MATCH WILL OCCUR. 

AFTER CLASSIFICATION, THE OUTPUT PATTERN OF THE HOPFIELD NET MATCHES 
BEST THE PATTERN OF DIGIT Two. 


Figure 11. Hopfield net response to the perfect input pattern 


The result was as we thought; it took the net only one iteration to recognize the 
input pattern as one of the M class stored patterns. From these simulations we can 
conclude that the Hopfield network when used as a classifier can be useful in a com- 
munication receiver where its task 1s to recognize the received bits, also to correct the 
corrupted ones and to recognize them. However, the Hopfield net is only iterating be- 
tween an input pattern and the ones that are already stored in the memory of the net. 
The number of these patterns (M) 1s a limitation to the proper operation of the net as 
a Classifier because of the convergence condition demonstrated bv Hopfield, which states 
that the net will converge with high probability if 4f<0.15N. A non-learning network 
is what the Hopfield net is, compared to other networks that we are going to describe 
in the following chapters. However, it has an important advantage over the others, its 
ability to recognize patterns even in noisy environement as long as the original pattern 


was stored in 1ts memory prior to its use, otherwise, a “no match” will occur. 
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Il. THE HAMMING NETWORK 


A. GENERALITIES : 

The Hopfield net, as we have seen, is often tested on problems of pattern recognition 
and classification by taking an input exemplar and reversing its bits randomly with a 
certain probability. The classifier in this study will calculate the Hamming distance to 
the exemplar of each class and select that class with the minimum Hamming distance to 
the specified input pattern. The Hamming distance is the number of bits in the input 
which do not match the corresponding exemplar bits. A net, which will be called the 
Hamming net, implements this algorithm using neural net component. Instead of cal- 
culating the Hamming distance directly, we will calculate N minus the Hamming dis- 
tance and maximize this function, where N is the number of elements or bits 1n a pattern 
representation. [Ref. 4] 

N minus the Hamming distance can be calculated from a weighted sum of the N 
elements of the input vector. If the elements of the input pattern to the net take on the 


values +1 and —1 for the respective states, then 





N= Nom = +) yx ea 
i=0 
where, 
x} 
Wy = > (3 — 2) 
and 
N 
G = 5 (3 — 3) 


Here x/ is the value of element 1 of the exemplar for class }. When all elements in the 
input vector match an exemplar exactly, each element in the sum of Equation 3-1 adds 
>. and adding c, given in Equation 3-3 gives a total of N. Whenever an element in the 
input pattern does not match the corresponding element in the exemplar, the prior total 


is decremented by 1 as required. [Ref. 4] 


Ig 


On the other hand, when elements of the input pattern x take on the values 0 and 
+ 1 for the —]1 and +] states, respectively, N. minus the Hamming distance can be cal- 


culated from: 


N—1 
N— Nam = + or Ca 
i=0 
where, 
+1 ifx = 41 
wy = bee 3 at 
-1 fx =0 
and 
N—1 
g=M=eNn-)a (3a) 
i=0 


In the above equation, 2 represents the number of zero elements in the exemplar 
for class j}. When all elements in the input pattern match an exemplar exactly, the sum 
in Equation 3-4 adds up to the number of positive input elements. This is added to the 
number of zero input elements results in N, as desired. The sum is reduced by one 
Whenever a zero input element that matches an exemplar becomes positive, and when- 
ever a positive input element that matches an exemplar becomes Zero. [Ref. 4] 

Here we have made a brief introduction to the Hamming net used as a classifier, as 
the net that calculates the Hamming distance to exemplars for all classes and then select 
that class which produces the minimum Hamming distance to the input pattern. Also , 
we have introduced two kinds of input patterns, one which elements take on +1 and 
—1 values, the other +1 and 0 values, for the +1 and —1 states, respectivelv. Its con- 
sequences on the calculation of N minus the Hamming distance were also introduced. 
The next paragraph will discuss the Hamming net in further detail and illustrate how the 


selection of the minimum is made. 


B. OPERATION OF THE HAMMING NETWORK : 
Two neural nets that are logically required to implement an optimum classifier for 


binary patterns will be assembled to form the Hamming net. One net forms the weighted 
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sum to calculate quantities related to the likelihood of the different classes. The second 
net picks the maximum. [Ref. 4] 
The first net that forms weighted sums is presented in Figure 12. An input pattern 


X is applied at the bottom of this net and an output pattern y is produced at the top. 





Figure 12. Feed-forward neural net used to calculate M weighted sums from the N 


elements of the input pattern [Ref. 4] 


The first laver of nodes sends values of the input pattern to the links feeding the 
second layer. The second laver of nodes uses nonlinear threshold logic elements to sum 
weighted values of the inputs and add internal offsets [Ref. 4]. Output values from the 
second layer are 

N=1 
y - é( 5 + > 0<j<M-1 6=7) 


i=0 


where, 


| 


a fa>0 
fie) = 40 a 3-8) 


In these equations, f(a) is a nonlinear function that models the nonlinearity inherent 
in a biological neuron. c, is an internal offset associated with each threshold logic node, 
and w, are positive or negative weights associated with the links [Ref. 4]. 

A number of different nets can be used to pick the maximum value from the y, out- 
puts of the feed-forward neural net shown in Figure 12. In situations where it is only 
important to know if the input matches a stored state verv closely, it is sufficient to 
identify those second-level nodes in Figure 12, with output values that exceed a specified 
threshold. This can be performed by modifying the constant c, added in Equation 3-7 
such that only the output of those nodes corresponding to closely matching stored states 


are positive. [Ref. 4] 


outputs 


HAH 
iS 
A 


Wy 





Figure 13. The iterative neural net called “maxnet” that picks the maximum of M 
inputs [Ref. 4] 








In the more general situation, a net must select the maximum over the .Vy, values. 
There are many different neural net structures which perform this task. A less complex 
net that uses feedback connections to pick the maximum output y, ( referred to as a 
maxnet ) is presented in Figure 13. [Ref. 4] 

Although this net 1s similar in structure to the Hopfield net (Figure 5), it uses 
threshold-logic nodes, relative to the threshold-logic nonlinear function described in 
Equation 3-8, rather than hard-limiting nodes and feeds the output of each node back 
to its input instead of disallowing this feedback path. The maxnet is a fully connected 
net made up of only M threshold logic nodes with internal thresholds set to zero. Input 
values are applied at time zero through the input nodes on the bottom of Figure 13. This 


initializes node outputs for each node at time zero [u,(0)] to the input values : 


u,(0) cs Jj J — Merah — Papa =f (3 ~ 9) 


The network then iterates to find the maximum using the following equation : 


u{t+1) = Af o4o —€ > 2A DES in) ly aes | (3 — 10) 
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In this equation, f, 1s the threshold logic function described in Equation 3-8. Each 
node inhibits all other nodes with a value equal to the node’s output multiplied by a 
small negative weight (¢) which 1s less than = Each node also feeds back to itself with 
unity gain. After convergence, only that output node corresponding to the maximum 
input will have a nonzero value. This value will generally be less than the original time 
zero value of that node. The output values of the net are thus simply the node output 


values after convergence : 


z= poo) j=0,1,...,—2,M— 1 (3— ty 


The maxnet will converge and find the maximum input when 


l 


Sivan (3 — 12) 


“di 
By convergence, we mean that the output nodes stop changing in time and only the 


output of one node corresponding to the maximum input is positive. Applying the 


threshold logic function f, on each one of these output nodes will result in only one 


ay, 


nonzero output node, the previous positive one, and a Zero value for all the others. The 


nonzero node corresponds to that exemplar class which best matches the input pattern. 
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Figure 14. |The complete neural network classifier referred to as The Hamming net 
[Ref. 2] 


The block diagram of the complete Hamming net, when used as a classifier, 1s 
completed by putting together the feed-forward neural net referred to as the lower sub- 
net (calculates the weighted sums) , and the maxnet referred to as the upper subnet (se- 
lects the node with the maximum output value). The complete Hamming net 1s then as 


shown in Figure 14. 


C. IMPLEMENTATION OF THE HAMMING NET : 
The operation algorithm of the Hamming net as a classifier can be described in four 
steps which the net must follow to classify a certain input pattern. The four steps of the 


al@oritimi are > mci) 








Step 1. Assign connection Weights and Offsets 


In the lower subnet: 


Wy = bi O=i:<N—1, 0s/s M-1 (3 — J3) 
and 
@, = = O<j<M-1 (3 — 14) 
In the upper subnet : 
+1, k=] 
Wn, K#l, &E< =, Oski<sM—I1 Ca 


In these equations, w, is the connection weight from input i to node j in the lower 
subnet and @ 1s the threshold in that node. The connection weight from node k to node 
lin the upper subnet is 4,, and all thresholds in this subnet are zero. x’ is element 1 of 


exemplar j . 
Step 2. Initialize with Unknown Input Pattern 
f= 
p,(0) = (Sous - 3 Ca — | (3 — 16) 


i=Q 


In this equation, y(z) is the output of node j in the upper subnet at time t, x, is ele- 
ment 1 of the input, and f/f, is the threshold logic nonlinearity. It 1s assumed that the 


maximum input to this nonlinearity never causes the output to saturate. 


Step 3. Iterate Until Convergence : using Equation 3-10, this process is repeated until 


convergence occurs after which the output of only one node remains positive. 
Step 4. Repeat by Going to Step 2 


First, weights and thresholds are set using the N elements of each one of the M 


stored patterns, as shown in Step | of the above algorithm. Then a binary input pattern 
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with N bits (N = 120 elements in this implementation) is presented at the bottom of the 
Hamming net at time zero. The N bits of the input pattern were chosen to take on the 
values +1 and —1 for +1 and —1 states, respectively. This input pattern must be pre- 
sented long enough to allow the lower subnet of the net to calculate matching scores 
which are going to be fed to the upper subnet (maxnet) allowing it to settle and initialize 
its Outputs. These matching scores are equal to N minus the Hamming distances be- 
tween the input and each one of the M exemplar patterns. This operation is done by 
using the equation given in Step 2 of the previous algorithm. The input is then removed 
and the maxnet iterates unt convergence using the iteration formula 3-10. By conver- 
gence, We mean that the output of onlv one node, corresponding to one of the M stored 
patterns, 1s a nonzero value. Classification is then terminated and the nonzero maxnet 
output node will point out the selected class that best matches the input pattern. 

The M stored patterns used in the implementation of the Hamming net when used 
as a Classifier, Were choosen to be similar to those used in the previous chapter; however, 
here we are using 10 exemplar patterns (MI = 10) instead of 8 used earlier. The 10 stored 
patterns (Figures 15 and 16) consist of 120 nodes (N=120) each as in the previous 
chapter (12 by 10 representation matrices). These 10 classes were choosen to represent 


all digits. 


Figure 15. The first four stored patterns 
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Figure 16. The rest of the 10 stored patterns 


D. SIMULATION OF THE HAMMING NETWORK: 

Using the Fortran program provided in Appendix B. we simulated the operation of 
the Hamming net when it 1s used as a classifier. The input pattern is choosen to be the 
digit “3° pattern which in the same fashion as for the Hopfield net, reversing its bits 
randomlv from +1 to —1 and vice versa with the same probability (0.25). 

The behavior of the Hamming net 1s illustrated in the output of the simulation 


ereerdim provided in Figure 17. 


THE UNKNOWN INPUT PATTERN TO THE HAMMING NETWORK, ON THE LEFT AS IM- 
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THE OUTPUT OF THE HAMMING NETWORK, WHERE EACH COLUNM REPRESENTS THE 
OUTPUT NODE VALUES FOR THE CORRESPONDING CLASSES AT A CERTAIN NUMBER 
OF ITERATIONS: 


NUMB OF ITERATIONS= 1 2 3 + > 6 7 8 9 10 
FOR CLASS 0: Ne, 6) 0 0 0 0 0 0 0 0 
FOR CLASS 1: ies 20 > 0 0 0 0 0 0 0 
FOR CLASS 2: Jol 18 é) 0 0 0 0 0 0 0 
FOR CLASS 3: 89 38 ZS 75h is 18 Vy 16 16 16 
FOR CLASS 4: 63 9 0 0 0 0 0 0 0 0 
FOR CLASS 75 22 7 1 0 0 0 0 0 0 
FOR CLASS 6: 65 12 0 0 0 0 0 0 0 0 
FOR CLASS 7: i> Ze 7 Ih 0 0 0 0 0 0 
FOR CLASS 8: 73 ay ues: 8 5 2 i 0 0 0 
FOR CLASae?: dd 25 ll 6 3 1 0 0 0 0 


CLASSIFICATION OF THE UNKNOWN INPUT PATTERN: 





THEN, THE DISTURBED UNKNOWN INPUT PATTERN TO THE HAMMING NETWORK 
CORRESPONDS TO THE PATTERN STORED OF THE CLASS THREE. 


Figure 17. Response of the Hamming net to the first input pattern 


For more clarity, we are going to illustrate the behavior of the Hamming net in the 
following plots. Each plot corresponds to the output of the net at the corresponding time 


step from time zero until its convergence. 
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Figure 18. The output of the Hamming net at t = 1 for digit “3” 
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The output of the Hamming net at t = 2 for digit ”3” 
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Figure 20. 
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The output of the Hamming net at t = 3 for digit “3” 
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Figure 21. 
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The output of the Hamming net at t = 4 for digit “3” 
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The output of the Hamming net at t = 5 for digit “3” 
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Figure 23. The output of the Hamming net at t = 6 for digit “3” 
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Figure 24. 
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The output of the Hamming net at t = 7 for digit “3” 
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Figure 25. The output of the Hamming net at t = 8 for digit “3” 


As vou can see, the net has effectively converged to the correct class corresponding 


to digit “3° and onlv 8 iterations were required to do so. By convergence, We mean that 
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the output nodes stop changing in time steps, and only the output of one node is a 
nonzero value. In our case, it was the output node of class 3. We say that the net has 
converged to the class that best matches the input pattern which is given by the position 
of the nonzero output node within the others. Further iterations will not change the 
output node values. 

The proof of convergence depends primarily on the fact the inhibition to the node 
containing the maximum value, in our case it was the node corresponding to class ‘3’, 
is always less than the inhibition to other nodes. This explains the fact that all output 
node values were decreasing on successive time steps. The output node values of those 
classes that are close to the input pattern were decreasing in time, but not in the same 
fashion as the classes that are totally different from the input pattern which were de- 
creasing faster. At convergence, the inhibition to the node with the maximum value re- 


auces to zero. [Ref. 4] 


inhib(t) =) ty wf) (3 — 17) 


iX/ 


where, inhib{z) is the second term of the right hand side of Equation 3-10, u(z) is the 
output of node i at time t and 7, 1s the inhibition weight between nodes in the upper 


subnet and it’s given by the following formula, 


tae i= J 
a — €, i¢# J, &€< al. O<ij<sM-] Cian!) 
M 

Node 3 corresponds to the maximum input, then on the first iteration, the inhibition 
to this node was less than the inhibition to all other nodes. This follows because all node 
outputs are positive and the sum of all outputs, excluding one in Equation 3-17, will be 
minimum when the maximum is excluded. Node 3 thus remains the maximum after the 
first iteration. By induction, it will remain the maximum over all iterations and it re- 
sponded as expected. [Ref. 4] 

The remainder of the proof of convergence depends on demonstrating that the out- 
put of node 3 is never driven to zero, but the outputs of all other nodes are. When 
Equation 3-12 is satisfied, inhib{z) is always less than the average value of all other node 
outputs. The inhibition to node 3 will thus be less than the average of the output of all 


nodes. Whenever a maximum exists, this inhibition will always be less than the current 
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Output of node 3 because the maximum of a set of positive numbers is always greater 
than the average. The output of node 3 will not be driven to zero while any other nodes 
have nonzero outputs. After all other node outputs are driven to zero, the inhibition to 
node 3 drops to zero, and the output of node 3 remains constant. The output of all other 
nodes will always be driven to zero because the inhibition to these nodes remains posi- 
tive on all iterations and approaches a positive constant as time increases. In practice, 
the maxnet will still converge and find the maximum when each weight w, is set to 
1 plus a small random component. This forces the net to find a maximum when 
the inputs to all nodes are identical. As a matter of fact, this discussion can be general- 
ized to all input patterns. [Ref. 4] 

Then, we presented another input pattern which is of the digit “9”. This pattern was 
noise corrupted in similar fashion as for digit “3”. The purpose of this simulation was to 
see if the net will behave as discussed earlier and if the number of iterations necessary 
for a successful convergence is function of the input pattern and how much noise dis- 
turbed it is. So, for the noisv pattern of digit “9”, the response of the net is as provided 
line higuKne 26: 

For more clarity, we are going to illustrate the response of the net in the following 
graphs for successive iterations. Each plot corresponds to the response of the net at the 
corresponding number of iterations, where the class output node values are decreasing 
from an iteration to the next. By inhibition, all of them will be driven to zero, some faster 
than others, except for the class output node corresponding to that to the stored 
exemplar that best matches the unknown input pattern. Then, this output node value 
will remain constant throughout future iterations while the zero valued output nodes 


will remain at zero. 


38 


THE UNKNOWN INPUT PATTERN TO THE HAMMING NETWORK, ON THE LEFT AS IM- 
POSED ON THE NETWORK AND ON THE RIGHT IN A MUCH CLEAR REPRESENTATION 
WHERE EVERY ( s ) REPLACES A 1 AND EVERY ( . ) REPLACES A -1: 


“1 1 21s+1i1#%i1é#%d1s21 41ei2 s os s s s £ 
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THE OUTPUT OF THE HAMMING NETWORK, WHERE EACH COLUNM REPRESENTS THE 
OUTPUT NODE VALUES FOR THE CORRESPONDING CLASSES AT A CERTAIN NUMBER 
OF ITERATIONS: 


NUMB. OF ITERATIONS= 1 Z s, 4 5) 6 y 8 2 10 
FOR CLASS 0: 48 0 0 0 0 0 0 0 0 0 
FOR CLASS 1: 58 10 0 0 0 0 0 0 0 0 
FOR CLASS 2: 60 I 0 0 0 0 0 0 0 0 
FOR CLASS 3: 74 ay IS: 9 5) 2 0 0 0 0 
FOR CLASS 4: 72 ZS WZ 6 A 0 0 0 0 0 
FOR CLASS 5: Fe 2D WA 6 2 0 0 0 0 0 
FOR CLASS 6: Sy 4 0 0 0 0 0 0 0 0 
BOK CLASS 7: 66 19 6 0 0 0 0 0) 0 0 
FOR CLASS 8: Sys: 10 0 0 0 0 0 0 0 0 
FOR CLASS 9: 90 45 34 30 28 Ze Zo 26 26 26 


CLASSIFICATION OF THE UNKNOWN INPUT PATTERN: 


Se eS re re em ee Se ee 
SS ee cr ee em ee 


THEN, THE DISTURBED UNKNOWN INPUT PATTERN TO THE HAMMING NETWORK 
CORRESPONDS TO THE PATTERN STORED OF THE CLASS NINE. 


Figure 26. Response of the Hamming net to the second input pattern 
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The output of the Hamming net at t = | for digit “9” 
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Figure 28. 
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The output of the Hamming net at t = 2 for digit “9” 
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Figure 29. 
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The output of the Hamming net at t = 3 for digit “9” 
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Figure 30. The output of the Hamming net at t = 4 for digit “9” 
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Figure 31. 
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The output of the Hamming net at t = 5 for digit “9” 
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Figure 32. 


is 02 oO} 
HIVIVA GQG0N LNdLNO SSVI9 


The output of the Hamming net at t = 6 for digit “9” 
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Figure 33. The output of the Hamming net at t = 7 for digit “9” 


These graphs have proven once more. the net’s mechanism of convergence described 


earlier. After 7 iterations. the net has effectively converged to the correct pattern of digit 
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“9”, which output node was the only nonzero node while all the others were driven to 
zero by the convergence process of the Hamming net. Then, seven iterations were used 
to converge to digit “9” in comparison to eight to recognize digit “3”. These simulations 
were for noise-corrupted input patterns, now we resimulated the net but this tme using 


Giewperfect patternvol digit “2% The response of the net is provided in Figure 34. 


THE UNKNOWN INPUT PATTERN TO THE HAMMING NETWORK, ON THE LEFT AS IM- 
POSED ON THE NETWORK AND ON THE RIGHT IN A MUCH CLEAR REPRESENTATION 
WHERE EVERY ( # ) REPLACES A 1 AND EVERY ( . ) REPLACES A -1: 
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THE OUTPUT OF THE HAMMING NETWORK, WHERE EACH COLUNM REPRESENTS THE 
OUTPUT NODE VALUES FOR THE CORRESPONDING CLASSES AT A CERTAIN NUMBER 
OF ITERATIONS: 


NUMB. OF ITERATIONS= 1 2 3 4 5 6 7 8 S 10 
FOR CLASS 0: 56 0 0 0 0 0 0 0 0 0 
POK CLASS 1: 66 9 0 0 0 0 0 0 0 0 
FOR CLASS 2: 120 67 54 49 47 4.6 45 45 4.5 45 
FOR CLASS 3: 84 ZS Ny ~ 0 0 0 0 0 0 
FOR CLASS 4: 64 7 0 0 0 0 0 0 0 0 
FOR CLASS 5: 84 28 12 “ 0 0 0 0 0 0 
FOR CLASS 6: 90 55 20 12 j 3 0 0 0 0 
FOR CLASS 7: 80 24 8 0 0 0 0 0 0 0 
FOR CLASS 8: Une NG! 6 0 0 0 0 0 0 0 
FOR CLASS 9: 54 0 0 0 0 0 0 0 0 0 


CLASSIFICATION OF THE UNKNOWN INPUT PATTERN: 





THEN, THE DISTURBED UNKNOWN INPUT PATTERN TO THE HAMMING NETWORK 
CORRESPONDS TO THE PATTERN STORED OF THE CLASS TWO. 


Figure 34. Response of the Hamming net to the perfect input pattern 
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Even for the perfect input pattern of digit “2”, the net took 7 iterations to success- 
fully converge to the correct response. So, the number of iterations is definitly not 
function of the input pattern and how noise disturbed it is but with how many stored 
patterns it shares many similarities (almost same distribution of +1 and —1). The num- 
ber of iterations necessary for a successful convergence depends on the output node 
values at the first iteration. The bigger magnitude these values have, the more similar 
their respective patterns are to the input and the more iterations will be necessary to 
drive them to zero by inhibition except for the correct output node. 

In our last simulation with digit “2” as a perfect input pattern, after the fourth iter- 
ation, only those output nodes corresponding to class “2” and class “6” were the only 
nonzero valued nodes. We can conclude, from this, that the input pattern, after the 
fourth iteration, is very close to the patterns of digit “6” and “2”, but more closer to “2” 
than “6” because of its higher output node value. The network took two more iterations 
to drive the output node of the “6” to zero. Then, the inhibition to node “2” drops to zero 
and the value of its output node remains constant. The input pattern 1s said to be digit 
- pattem: 

This discussion can be generalized to explain the previous behavior of the network 
to digit “3” and digit “9” as input patterns. In all cases, the output node with the higher 
magnitude, after the first iteration is completed, is always the correct node but the con- 


vergence mechanism of the net is not to make decisions at this stage. 


4§ 





IV. THE CARPENTER / GROSSBERG NET 


A. GENERALITIES : 

Classified as a self-organizing neural net, the Carpenter / Grossberg net self- 
organizes and self-stabilizes its recognition codes in response to arbitrary sequences of 
binary input patterns. Top-down attentional and matching mechanisms are critical in 
self-stabilizing the code learning process. The architecture embodies a parallel search 
scheme which updates itself adaptively as the learning process proceeds. After the 
learning process has self-stabilized, the search process is automatically disengaged. 
Thereafter, input patterns directly access their recognition codes without any search. 
Thus, recognition time does not grow as a function of code complexity. 

A novel input pattern can directly access a category if it shares invariant properties 
with the set of familiar exemplars of that category. These invariant properties emerge in 
the form of learned critical feature patterns, or prototypes. The architecture possesses a 
context-sensitive self-scaling property which enables its emergent critical feature patterns 
to form. They detect and remember statistically predictive configurations of featural el- 
ements which are derived from the set of all input patterns that are experienced. Four 
tvpes of attentional processes (priming, gain control, vigilance, and intermodal compe- 
tition) are mechanistically characterized. Top-down priming and gain control are needed 
for code matching and self-stabilization. Attentional vigilance determines how good the 
learned categories will be. If vigilance increases due to an environmental disconfirma- 
tion, then the svstem automatically searches for and learns the best recognition catego- 
ries. [Ref. 6] 

This chapter develops a theorv of how recognition codes are self-organized by a 
class of neural networks whose qualitative features have been used to analvse data about 
speech perception, word recognition and recall, visual perception, olfactory coding, 
evoked potentials, thalamocortical interactions, attentional modulation of critical ter- 
mination, and amnesia. These networks comprise the adaptive resonance theorv (ART) 


characterized as a system of ordinary differential equations. 


B. IMPLEMENTATION OF THE CARPENTER / GROSSBERG NET: 
The neural network that will be discussed in this chapter is known as an ART sys- 
tem, after the adaptive resonance theory introduced by Grossberg [Ref. 7], see Appendix 


C. Recently, ART networks have been further studied and their dynamic properties 
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have been derived in a series of theorems. These theorems predict both the order of 
search, as a function of the learning history of the net, and the asymptotic category 
structure self-organized by an arbitrarv binary input sequence. 

The operation of the ART system discussed in Appendix C will be used to develop 
a neural net Known as the Carpenter / Grossberg net, using neural net components, 
which will form clusters and is trained without supervision. The net can learn from input 
patterns and later differentiate between new and stored (learned) patterns. If the new and 
unknown input pattern is classified as a previously learned pattern at a certain level of 
vigilance, it will be ignored, but if it is not, it will be added as a new pattern by the net. 
This process is repeated for all input patterns. The number of learned patterns thus 
grows with time and depends stronglv on the level of vigilance (threshold) used to com- 
pare input patterns to the already stored ones. 

The operation of the Carpenter / Grossberg net which forms clusters (learned pat- 


terns) and is trained without supervision is given in eight steps : [Ref. 2] 


Step 1. Initialization 


4,(0) = 1 (4—1) 


l 


T+ ae 


b,(0) = 


O<i<gN-1 ,O0S/j<sM-1 


Set pi, ee = ae 

In these equations b,(¢) is the bottom-up and ,(z) is the top-down connection weight 
between input node i and output node j at time t as shown in Figure 35. These weights 
define the exemplar specified by output node j. The fraction p is the vigilance threshold 


which indicates how close an input must be to a stored exemplar to match. 
Step 2. Apply New Input 


Step 3. Compute Matching Scores 
N=1 


w=) bfx, OS/<SM-1 (429 
= 
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In this equation y, is the output of node j and x, is element i of the input pattern 
famch can be Q or t. 


QUTPUT 





Figure 35. The major components of the Carpenter / Grossberg classification net 
[Ref. 2] 


Step 4. Select Best Matching Exemplar 


y= as {1} (4-4) 


This is performed using extensive lateral inhibition as in the maxnet. 


Step 5. Vigilance Test 





a 
IM = > x, (4-5) 

| t=) 

| N=] 
IT*M = Days (4-6) 
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| T* x 
is ——— > 0? a 
be : Se 


If YES then GO TO Step 7, otherwise GO TO Step 6 


Step 6. Disable Best Matching Exemplar 
The output of the best matching node selected in Step 4 is temporarily set to zero 
and no longer takes part in the maximization of Step 4. Then go to Step 3. 


Step 7. Adapt Best Matching Exemplar 


y(t +1) = iy(2) x; (4 — 8) 


(4a 


Step 8. Repeat by Going to Step 2 
(First enable anv nodes disabled in Step 6) 


After initialization of the net and presentation of an unknown input pattern, 
matching scores are computed using feed-forward connections. The node corresponding 
to the exemplar with the highest matching score is selected using lateral inhibition 
among the output nodes as in the maxnet (Hamming net), where each output node 
corresponds to a stored exemplar. This net differs from the Hamming net in that feed- 
back connections are provided from the output nodes to the input nodes and elements 
of both inputs and stored exemplars take on only the values 0 and 1. 

The selected exemplar, from the highest matching score, is then compared to the 
input by computing the ratio of the dot product of the input and the best matching 
exemplar (number of 1 bits in common) divided by the number of 1 bits in the input. If 
the ratio 1s greater than a threshold value (vigilance) which was set at the initialization 
of the net (Step 1 of the algorithm), then the input is considered to be similar to the best 
matching exemplar and that exemplar is updated bv performing a logical AND opera- 
tion between its bits and those in the input. On the other hand, if the ratio is less than 


the vigilance threshold, the output node with the highest matching score 1s temporarily 
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set to zero, disabled by provided mechanisms. The same input pattern is presented again 
to the net for another test. The cycle continues until one stored exemplar matches the 
input or this pattern is considered to be different from all stored exemplars and it is 
added as a new one. Generally, when the first test fails the input is a new exemplar. Each 
additional exemplar requires one output node and 2N connections to compute matching 
Scores. 

The vigilance threshold, which ranges between 0.0 and 1.0, determines how close a 
new pattern must be to a stored exemplar in order to be similar. A value near 1.0 means 


a close match is necessary and smaller values accept a poorer match. 


C. SIMULATION OF THE CARPENTER / GROSSBERG NET : 

Using the Fortran program provided in Appendix D, We simulated the behavior of 
this net. For this simulation, a vigilance threshold of 0.9 was choosen, which means that 
an input pattern must be very close to a stored exemplar to be considered similar. The 
patterns used in this simulation were of the letters “C”, “E”, “F” and were choosen to be 
of 64 element representation (matrices of 8 by 8). In all the figures provided in this 
discussion, we have made a black pixel to correspond to an element of value 1 and the 
white pixel for the value of 0. The actual input patterns used are provided in Appendix 
D. 

Initially, the storage memory of the net was empty. To train this net, an input pat- 
tern representing the letter “C” was presented first and 1t was automatically stored as the 
net starts to learn. Now internal connection weights of the net are altered to form an 
internal exemplar that is identical to the letter “C” and we have the first output node of 
the net. In the same fashion, every learned pattern will be stored as 2N connection 
Weights and one output node is added to the net. These 2N connections weights will 


form an internal exemplar for the respective pattern stored. 
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In the storage memory of the net, we have only one stored exemplar : 


After, an input pattern representation of the letter “E” was applied. The response 


of the net to this input was : 


BECAUSE, THE RATIO IS LESS THAN THE VIGILANCE THRESHOLD 
THE INPUT PATTERN IS CONSIDERED TO BE DIFFERENT FROM 
ANY EXEMPLAR PATTERN STORED. THIS INPUT PATTERN IS 

THEN STORED WITH THE OTHERS AS A NEW EXEMPLAR PATTERN. 


Here “E” was compared to “C” as described 1n Step 5 of the clustering algorithm of 
the net and since the ratio was less than the vigilance threshold we now have two stored 


exemplars. 


For a new input pattern representation of the letter “F”, the response of the net was 
the same as for “E”. Here the input pattern was compared to both stored exemplars, but 
at all times the ratio was less than the vigilance threshold. The input pattern of “F” is 


then added as a new exemplar leading to three stored patterns. 
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At this point, we will try something different. We will present a noisy version of the 
letter “F” with a missing black pixel in the upper edge as shown in Appendix D. The 


reaction of the net was: 


BECAUSE, THE RATIO IS GREATER THAN THE VIGILANCE 
THRESHOLD, THE INPUT PATTERN IS CONSIDERED TO 
MATCH A STORED PATTERN WHICH IS UPDATED BY 
PERFORMING A LOGICAL 'AND' OPERATION BETWEEN 

ITS BITS AND THOSE OF THE INPUT PATTERN, AND 

THE NEW UPDATED PATTERN WILL LOOK LIKE: 


In this part of the simulation, the input pattern is found to match the stored 
exemplar of “F” because the ratio of the vigilance test was found greater than the vigi- 
lance value, i.e., the two patterns have many elements in common. The result was a 
degraded “F” due to the AND operation performed during the updating. Now in the 
memory of the net we still have three patterns with some changes in the pattern of the 
lemeer [| : 


Presenting an even more noisier version of the pattern of “F” given in Appendix D, 


the reaction of the net was: 


BECAUSE, THE RATIO IS LESS THAN THE VIGILANCE THRESHOLD 
Mee INPUT PATTERN IS CONSIDERED TO BE DIFFERENT FROM 
ANY EXEMPLAR PATTERN STORED. THIS INPUT PATTERN IS 
THEN STORED WITH THE OTHERS AS A NEW EXEMPLAR. 


Here the input pattern was compared first to the stored pattern of the noisy “F”, but 


the ratio was less than the vigilance value. Then it was compared to the other stored 
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exemplars, one in each cycle, but the ratio was still less than the vigilance value. The 
input pattern is then considered different from existing exemplars and it is added as a 


new one in the memorv of the net. At this point, we have four stored patterns : 


These results illustrate the inaccuracies of this net in a noisy environement. For a 
vigilance value of 0.9, we took a stored pattern and changed some of its elements simu- 
lating the presence of small amount of noise in the channel. Then, presenting it again 
as an input pattern has made the net take it as a new pattern to be stored with the cor- 
rect version. Besides the noise, the value of the vigilance test can also alter the behavior 
of the net as we are going to show. Using the same input pattern sequence as before, 
we are going to simulate the clustering algorithm of the net, but this time with 0.7 as the 
vigilance threshold. Starting by presenting the pattern of the letter “C” as the input, the 
net automatically stored it in its empty memory as the net starts to learn. Now internal 
connections weights of the net are altered to form an internal exemplar that is identical 
tO thelelicns 


In the storage memory of the net, we have only one stored exemplar : 


Then, an input pattern representing the letter “E” was presented to net. The response 
of the net was: 


BECAUSE, THE RATIO IS GREATER THAN THE VIGILANCE 
THRESHOLD, THE INPUT PATTERN IS CONSIDERED 

TO MATCH A STORED PATTERN WHICH IS UPDATED BY 
PERFORMING A LOGICAL 'AND' OPERATION BETWEEN 

ITS BITS AND THOSE OF THE INPUT PATTERN, AND 
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THE NEW UPDATED PATTERN WILL LOOK LIKE: 


Here, comparing the input pattern “E” to the stored “C” as described in Step 5 of the 
clustering algorithm. The ratio was found greater than the vigilance value (0.7). The re- 
sult was a degraded “C”, as shown, due to the AND operation performed on its bits 
during the updating. In the memorv of the net, the degraded pattern of “C” 1s stored 
instead of the initial pattern. Now we still have only one stored pattern. 

For the pattern of the letter “F” as an input, the net responded with the following 


message: 


BECAUSE, THE RATIO WAS LESS THAN THE VIGILANCE THRESHOLD 
THE INPUT PATTERN IS CONSIDERED TO BE DIFFERENT FROM 
ANY EXEMPLAR PATTERN STORED. THIS INPUT PATTERN IS 

THEN STORED WITH THE OTHERS AS A NEW EXEMPLAR. 


After comparing the input pattern of “F” to the stored and degraded pattern of “C” 
as in the vigilance test, the ratio was found to be less than the vigilance threshold which 


results in another stored pattern : 


Now, we are going to present the noise corrupted pattern of “F”. The net’s response 
was: 


BECAUSE, THE RATIO IS GREATER THAN THE VIGILANCE 
THRESHOLD, THE INPUT PATTERN IS CONSIDERED 
TO MATCH A STORED PATTERN WHICH IS UPDATED BY 


ay) 


PERFORMING A LOGICAL 'AND' OPERATION BETWEEN 
ITS BITS AND THOSE OF THE INPUT PATTERN, AND 
THE NEW UPDATED PATTERN WILL LOOK LIKE: 


The matching pattern was of the letter “F”. At this point, checking the net’s memory 


will reveal the storage of the two patterns : 


Presenting a more corrupted pattern of “F”, the response of the net was as before. 
The input pattern was found similar to the first corrupted version of “F” (the ratio was 
greater than the vigilance threshold). The stored pattern of the corrupted “F” was again 
more disturbed after the AND operation was performed between its bits and the input 
pattern. The result, once again, replaced the previous version of the corrupted “F” in the 


memory of the net: 


The results of the two simulations show clearly how the noise and the vigilance 
threshold can affect the performance of the Carpenter / Grossberg net. We have seen the 
net performing well for perfect input patterns and when adding a small amount of noise 


it behaves totally different. With no noise, a lower vigilance value can make the net 
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consider two different patterns to be similar. We have seen this in the second simulation 
when we have presented the pattern “E”, which was mistakenly considered similar to “C” 
for 0.7 vigilance. On the other hand, a higher vigilance threshold can make the net con- 
sider two patterns, which are most similar, to be different. Thus, this net should not be 
used in a noisy channel with a high vigilance value; otherwise, the number of stored 
patterns will grow rapidly in time as input patterns are continuously presented until all 
available nodes are used up. A proportional adaptation of the vigilance threshold to the 
existing noise in the channel can make the net to perform perfectly during training and 


testing. 
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V. NEURAL NETWORK AS A BINARY MAXIMUM-LIKELIHOOD 
SEQUENCE ESTIMATOR 


A. GENERALITIES : 

Bandwidth-efficient data transmission over telephone and radio channels is signif- 
icantly improved by the use of adaptive equalization to compensate for the time 
dispersion introduced by the channel. 

During the last two decades, a steady research effort has produced a rich body of 
theory 1n the field of adaptive equalization and the more general field of adaptive re- 
ceivers. From this work, a class of nonlinear receivers referred to as maximum-likelihood 
sequence estimation receivers have emerged as front-runners with respect to error rate 
performance. However, the high degree of computational complexity of the optimal 
maximum-likelihood receivers has prohibited their use in many applications. It will be 
shown that neural networks can be used to implement the maximum-likelihood sequence 
estimation and that the networks offer an attractive alternative for implementation. 
[Rets) 

Intersymbol interference caused by the bandlimiting effect of the channel 1s re- 
viewed. A maximunn-likelihood receiver designed to detect data symbols in the presence 
of intersymbol interference and additive Gaussian noise is considered and the theory 
behind maximum-likelihood sequence estimation 1s reviewed. 

The maximum likelihood sequence estimation function 1s mapped onto a neural 
network structure. A neural network based receiver structure will be described which can 
be used for stationary or time-varving channels. The MLSE neural network will be sim- 


ulated on the Mainframe and some results of its simulation will be presented. 


B. MAXIMUM-LIKELIHOOD SEQUENCE ESTIMATION : 
Consider a baseband synchronous data communication link used to transmit a se- 


quence of numbers called data or information symbols, denoted by 
{ey Gas Gs By oJ 


The symbols are independent and can, with equal probability, be either +1 or —l. 


Let M be the number of data symbols in a transmitted sequence and assume 
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transmission starts at time t = 0 and ends at tmet = MT. The receiver will observe the 


signal y(t) during the time interval starting at t = 0 and ending atr = ¢,, where 


ie CV tae) T 


where L is the channel memory in units of T. 
Denote the time inteval 0 > ¢, by J,. By its definition, a maximum-likelihood re- 
ceiver determines {a,} as the best estimate sequence {a,} that maximizes the likelihood 


function pL y(t), te f, | {a,}] given by 


Ltd, 
pLy(y), re ipl {a,}] ~ ¢€xXp ea I I nt; | {a,}) Ky (t ns fy) n(t | {a,}) at, ash - 1) 


where K7'(r) is the inverse of the noise autocovariance function K,(r) and 


M 
n(t| {a,}) = 0) — ay h(t — KT) (5 — 2) 
k=l 
where h(t) is the impulse response of the matched filter used in the adaptive maximum- 


likelihood receivers (Figure 36). Rearranging Equation 5-2, 


M 


y(t) = > ay h(t — KT) + nl | {a,)) (5-3) 


k=] 


A block diagram of an adaptive maximum-likelihood receiver for the data trans- 


mission model described by Equation 5-3 is illustrated in Figure 36. 


Peete: | J| MAXIMUM-LICELITOOD 
Be SEQUENCE ESTIMATOR 


COEFFICIENT 
ESTIMATOR 





Figure 36. Adaptive Maximum-Likelihood Receiver [Ref. 3] 
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The impulse response of the matched filter in Figure 36, designed to improve the 


signal-to-noise ratio, 1s given by 
g(t) = A(-1)* Ky () (5-4) 


where * denotes the convolution function. Substituting Equation 5-2 into Equation 5-1, 


expanding the terms in the braces and considering only terms that depend on {a,}, vields 


M MM 
pln(s) | {a,}] ~ exp 28 i ) > GS af (5-5) 
i= i=] k=] 
Wene. 
n= | | Ry — 2D)" (q — ) Xe) de de (5-6) 
feet 
Ss, = | | Re — iT) K' (1, — 4) h(n — KT) dt, de 
| eae | 2 
fat, (S- 1) 
and 
A(t) = A(-0) (5 9 


The quantities z, and s, can be interpreted as sample values taken at the output of 
the matched filter, where z, is obtained by sampling the output z(t) of the matched filter 
once everv T seconds and s, ‘s account for the combined response of the transmission 
channel and matched filter. The s,s are symmetric and s, = 0 for |/| > L. [Ref. 8] 

Under maximum-likelihood criteria, the estimated sequence is that for which ex- 
pression 5-5 is maximized. Since 5-5 is monotonically increasing function of the term in 


braces, given by 


M MM 
Jydia,}) = D2 a; 2; — Sa Sik Uke (5 — 9) 
i=) 


i 
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maximizing Equation 5-5 is equivalent to maximizing Equation 5-9. The notation 
J,{{a,}) indicates the cost function for the sequence a,, a, ..., Ay. Equation 5-9 will be 
referred to as the MLSE cost function. [Ref. 3] 

The estimation procedure using direct evaluation of the MLSE cost function re- 
quires that Equation 5-9 be evaluated for all the possible sequences of length M that can 
be formed from data symbols +1 and —1. Thus Equation 5-9 must be evaluated 24 
times to obtain an estimate of the sequence {a,}. To perform the estimate in real time, 
which is required by most communication links, the 2“ computations of Equation 5-9 
must be performed in MT seconds. In most cases, direct evaluation of ti.z MLSE cost 
function is too computation intensive to be of practical use. [Ref. 3] 

The number of computations required can be greatly reduced bv the use of the 
Viterbi algorithm [Refs. 8,9], which requires on the order of 24-! multiply-and-add oper- 


ations during each signaling inteval T. 


C. NEURAL NETWORK: 

Any neural network has, as discussed before, parallel input channels, parallel output 
channels and a large amount of interconnections between the neural processing ele- 
ments. Figure 37 illustrates the general structure of a Hopfield neural network. The 
processing elements (nodes), or neurons, are modelled as amplifiers in conjunction with 
feedback circuits comprised of wires, resistors and capacitors organized so as to model 
the most basic computational features of neurons, namely axons, dendritic arborization 
and synapses connecting the different neurons. [Ref. 10] 

The model considered here for implementation of the MLSE neural network is that 
due to Hopfield and Tank [Ref. 3]. 

The amplifiers have sigmoid monotonic input-output relations, as shown in Figure 
37. The function v(t) = g [u(r)] which characterizes this nonlinear input-output relation 
describes the output voltage v(t) due to an input voltage u(t). The time constants of the 
amplifiers are assumed negligible. However, like the input impedance caused by the cell 
membrane in a biological neuron, each amplifier j has an input resistor p, leading to a 
reference ground and an input capacitor c, These components partially define the time 
constants of the neurons and provide for integrative analog summation of the synaptic 
input current from other neurons in the network. 

In order to provide for both excitatory and inhibitory synaptic connections between 
neurons while using conventional electrical components, each amplifier is given two 


outputs, a normal (+) output and an inverted (—) output. 
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Figure 37. Hopfield Neural Network [Ref. 3] 


The minimum and maximum outputs of the normal amplifier are taken as 0 and 1, 


while the inverted output has corresponding values of 0 and —1. A synapse between 
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neurons is defined by a conductance w, which connects one of the two outputs of am- 
plifier j to the input of amplifier i. This connection is made by a resistor of value 
Ee — ] 


4 | W,,| 
normal (+) output of amplifier j. For an inhibitory synapse (sw, < 0), itis connected to 





. If the synapse is excitatory (w, > 0), this resistor is connected to the 


the inverted (—) output of amplifier j. The matrix w, defines the connectivity among the 
neurons. The net input current to anv neuron i (and hence the input voltage u,) 1s the 
sum of the currents flowing through the set of resistors connecting its input to the out- 
puts of the other neurons. [Ref. 9] 

The set of differential equations describing the dynamics of the neural network 


shown in Figure 37 with M neurons are given by 


Af 


du,(t) ult 
G— = 2, nl) ~ Ro th fal M (5 — 10) 








where «(7) = g,([uz)] and R, 1s the parallel combination of p, and the K, ’s, and C, is the 


capacitance of amplifier }. 


1 I I : 5 
——— = os _ ee i= 2 eee gy M e— 11) 
R, Pi mR 


momsinplicity, we assume that g[.] = e[.]. k, = R and C, = C, independent of i. 








Dividing Equation 5-10 by C and redefining w, = = ania = ra the equations of 
motion become : 
dur) - (r) 
i U; 
—— = en) -— “th i= 1M (5 — 12) 
k=] 


where t = RC 1s the time constant of the circuit. [Ref. 9] 

In the Hopfield net operation, it was shown that the equations of motion for a net- 
work with symmetric connections (w, = w;,) always lead to a convergence to stable 
states, in which the outputs of all neurons remain constant. Also, when the width of the 
amplifier gain curve in Figure 37 is narrow, the stable states of a neural network com- 


prised of \{ neurons are the local minima of the quantity 


M A 


E= - duh - > DDO me lO (5-13) 


i=] i=] k=) 


When high amplifier gain is used, the minima occur only at the corners of an M- 


dimensional hypercube defined by v, = +1 or —1. 


D. MAPPING OF MLSE ONTO A NEURAL NETWORK : 
Maximizing the MLSE cost function described by Equation 5-9 is equivalent to 


minimizing the following expression 


M MM 


Fy ({a,}) = »2 a,z; + »>4 S)_, a (5 — 14) 
[=1 


i=1 k=] 


Where a,'s (of only +1 and —1) values which minimize Equation 5-14 are unknown. The 
z,s and s_,s are known. Comparing Equations 5-13 and 5-14 and equating variables 


as follows , 
— zy — e — s 
2 Z; = if ; ey Sik = Wir 5 a; = v(t) 


reveals that the two expressions are identical under these substitutions. From Equation 


5-7, we recall that's = sSmeenercio:: 


: = =) raat one. i: ; 
Wip = 25,4 = —2 Sy) = Wy; 


which satisfies the svnaptic interconnection symmetry condition. The synaptic intercon- 
nections for the neural network are determined by the coefficients which describe the 
combined response of the channel and matched filter. Let W denote a matrix of synaptic 
connections w,. Then the svnaptic connection matrix W for the network is given in 
Figure 38. 

The externally supplied input current for each neuron, /,, is determined by observa- 
tion z,, 1<i< Af. With the input voltage u(r) initially at zero, the input sequence is 
applied to the network. After the network settles the estimated sequence {a,} is read from 
the output of the neural amplifiers. A diagram of the MLSE neural network 1s shown in 
Figure 305m helo) 
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Figure 38.  Miatrix of synaptic connections [Ref. 3] 


NEURAL NETWORK 
INTERCONNECTIONS: Wya's 





Figure 39. MLSE neural network [Ref. 3] 


This development assumed that the transmission channel is stationary. which 1m- 
plies that the s,s describing the combined channel and matched filter response do not 
change with time. Often, this is an unrealistic assumption. [Ref. 3] 

The MLSE cost function given by Equation 5-14 can be written for the time-varving 


channel as 
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MoM 


Tae >y, am + >) as ya (5 — ie 


i=) k=1 


From Appendix E and using Equation 5-15, the parameters for the MLSE neural 
network are given by [Ref. 3] 


22, = aS a(S oe ar Sok) = Wiz, a = V2) (5 =a) 


FE. NEURAL NETWORK MAXIMUM-LIKELIHOOD RECEIVER : 

A block diagram of the adaptive maximum-likelihood receiver incorporating the 
neural network for MLSE 1s shown in Figure 40. Registers R,, R,, ..., Ry forma shift 
register used to store the M observations. With all amplifier inputs u., i = 1, ..., M, 
initially at zero, switches SIF,, i = 1, ..., Mf, are simultaneously closed and the network 
is allowed to settle. The output of each neural amplifier 1s applied to the input of a de- 
cision device which outputs a +1 or —1 for a positive or negative input respectively. 
Once the network has settled, the estimated sequence is read at the output of the deci- 
sion devices as shown in Figure 39. [Ref. 3] 

In some cases, the length of the network, M, will be considerably less than the total 
number of data svmbols in a transnutted sequence. For example, suppose the transmit- 
ted sequence consists of K x .Wf data symbols. one approach 1s to load the first set of M 
observations and estimate the corresponding data symbols. After the estimate is ob- 
tained. the second set of observations would be loaded and the second set of data sym- 
bols estimated. The procedure would be performed a total of K times to obtain an 
estimate of the entire transmitted sequence. The primary problem with this approach 1s 
that it does not take into account the truncation of the observation sequence. [Ref. 3] 

The effect of the truncation can be described by considering the role of the obser- 
vations in the estimation. Let z, 1<k< M, denote an observation applied to the ex- 
ternal input of neural amplifier k, where z® is the i“ observation from the received 


sequence. 
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Figure 40. Neural Network Based Maximum-Likelihood Receiver [Ref. 3] 





Since the channel memory is L, all information concerning the identity of data 


svmbol a, is contained in observations: [Ref. 3] 
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(kK-L)  _(k-L+1) _(k+L) 
fj, 9 2pof4) 3 eee 


for k in the interval 
L<k <M-L+1 (5 — 17) 


All the observations containing information about the data symbol estimated by 


neural amplifier k are available to the network. On the other hand, for k in the intervals 
l<k <lL & M-L+iskewm™M (5 — 18) 


some of the observations containing information about the data symbol estimated by 
neural amplifier k are not available to the network. Therefore, one would expect more 
errors to occur in estimates a” for k in the interval given by Equation 5-18 than in the 
interval given by Equation 5-17. [Ref. 3] 

This problem can be solved by overlapping the sequences used for each estimation 
iteration. Assume a set of M observations have been received and the network has 
produced a set of M data symbol estimates. Rather than accept all M estimates as valid, 
onlv estimates from neurons L+1 through M-1 are taken as valid. This, of course, cor- 
responds to the estimates based on complete information about the symbols being esti- 
mated. From this set of observations, the observations in shift registers M, M-l,..., p+ 1 
are saved, where 2L < p< J —1. A new set of p observations are shifted into the shift 
registers and the network performs another estimation. Essentially, this procedure 
amounts to shifting in p rather than M new observations after each estimation cycle. 
[Ref. 3] 


F. SIMULATIONS AND RESULTS : 

The neural network based MLSE receiver structure was implemented and simulated 
on the Mainframe. The program used 1s a self-driving program, provided as Appendix 
F. The network was simulated by numerically solving the set of M differential equations 
of Equation 5-12. The differential equations solver used in the simulation was the sub- 
routine DGEAR of the IMSL hbrarvy. 

Then the M output values of DGEAR subroutine were passed through their re- 
spective neural amplifiers. The input-output function of the neural amplifiers was im- 


plemented as a hyperbolic tangent function. 


(2) = — tanh[Gu()] i = 1, .., Af (5 — 19) 
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where G is the gain constant. Increasing G increases the slope of the input-output curve 
in the transition region and reduces the width of the region (see Figure 37) [Ref. 3]. 
The transmission channel impulse response 1s modeled by a finite response square 


cosine function given by !Ref. 10] 


if 
F(a) aS ae le) a 

h(t, 0) = ° , iG— 20) 
0, a 


where the multiplicativeterm F(a) is included to model the time-varving channel. For the 
stationarv channel simulations, F(a) 1s taken as 1. Channel interference also includes 
additive White Gaussian noise n(t). The combined response of the channel and matched 
miter Js then (Ref. 10] 


F*(o) Qn 3, 
: Ty — |r| 1+ cos eo SI te) fn) pee el 
" =| ay ‘ : If : - z sin(2nle1To)¢ Hel < To, 


0 Mime 1p 











Where NV, denotes the single-sided spectral density of the additive White Gaussian noise 
n(t), and 7, 1s the time duration of the intersymbol interference. Equations 5-20 and 5-21 
are sampled at intervals of T seconds, where T is the bit duration, to generate the L + 
] discrete time channel coefficients (i(a)’s) and 2L + 1 discrete time coefficients de- 
scribing the combined response of the channel and matched filter (s(a)’s). WLSI imple- 
mentation using sequential processing techniques have been reported for data rates up 


to 2400 bits’second [Ref. 3]. The channel memory L is given by 
L=— (S — 22) 


T ; ae ; 
Actually, L is the largest integer less than or equal to =a but in this simulation we 


are going to take L as = 
Using the coefhicients generated by sampling Equations 5-20 and 5-21 and assuming 
a stationary channel (F(a) = 1) and baseband transmission model, the received samples 


(y,s) are generated by the expression 
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M 


yb = » hyp 35 n; oo ly pees rie (5 2 
| 


where ), replaces y(t) att = 11, 4_, replaces h(t) att = (1- k)T and n, is a sample of the 
additive White Gaussian noise n(t) at t = iT. For this simulation,the noise samples are 
generated by a Gaussian random number generator GGNML of the IMSL library. The 
data svmbols (a,’s) which are +1 and —1, are generated with equal probability using a 
uniform random number generator GGUD of the IMSL library. Then, the observations 


(z, S) are generated bv the expression 


M 
Z,= Vp Nps, nN = Aiea (S — 24) 
k=1 


The s,’s and z,’s generated by the Equations 5-23 and 5-24 are substituted into 
Equation 5-16 to define the parameters of the neural network. A gain factor G of 10000 
was used for the simulations because of the very small output numbers of the differential 
equations solver DGEAR. Each simulation started with zero initial conditions and the 
computations Were stopped after simulation of 5t seconds (5 time constants). The esti- 
mated sequence, the M outputs of the neural amplifiers (Figure 39), was then compared 
to the transmitted sequence and the total number of errors were recorded. Also, the es- 
timated sequence was compared to an estimate obtained by direct computation of the 
MLSE cost function and the number of data symbols which differed between the two 
estimated sequences was recorded. 

Table | lists the number of neurons (M) used in the simulation, the network time 
constant (t), the channel memory L which was taken to be 2 for all the simulations, the 
signal-to-noise ratio (SNR) given at the output of the matched filter and is computed 
bv 


ce 


SNR = AN, 





the number of data (p) shifted into the registers at each simulation step, the number of 
symbols transmitted (N), the error data for each simulation. The last column of Table 
1 lists the number of data svmbol estimates which differed between the neural network 


estimates and direct computation of the MLSE cost function. 


12: 


No. of ; Total 
Meirons | COD: symbols Total aie 


(M) stant transmitted | Errors 
(msec) 


Ts [7 [2 | 8s [6 | 20 | o | 0 
To Pus [2 | 2 [6 | 20 | o | 0 
[9 far | 2 [| 16 [6 | 200 [0 | 0 
Ts fas | 2 | 0» | «6 | 200 [0 [0 


ences 
(N) 


ar pas 2 fs | 2 | 100 [0 [0 
arf ss | 2 | 2 | 2 | 10 [0 [0 
Tar pes | 2 fo | 2 | 100 [0 [0 
a 
[3s fos [2 [8s | as [100 | 0 [0 
Tas [6 | 2 | 16 | as [100 [0 [0 
[as fos | 2 | 2» | is [100 [0 | 0 


Table 1. SIMULATIONS RESULTS FOR MLSE NEURAL NETWORK (STA- 
TIONARY CHANNEL) 





To simulate the time-varving channel, the multiplicative F(~) will be changed at each 


sampling instant. The value of F(«) is constrained to be in the interval 


0.8 < F(a) < 1.0 


The random number generator GGUD of the IMSL library was used to generate 
uniformly distributed samples, A, as described in the computer program of Appendix 


F. The generated samples A, are distributed in the interval 


0.1 < A, < 01 


So at a certain sampling instant, say a = iT, the value of F(~) is computed by 


FUT) 


0.9 + A, 


The transmission channel impulse response samples /{)’s and the combined response 
of the channel and matched filter s(’s along with z, are the parameters describing the 


time-varying channel at a certain time t = iT. With the exception of this modification. 
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the simulator for the time-varving channel is identical to that for the stationary channel. 


Results of this simulation for the time-varying channel are listed in Table 2. [Ref. 3] 


NO.OF 


No. of, symbols Total 


neurons 


(M) 7 “S* Errors 


ty Prop ty J to 


tJ 





Tablewz: SEMULATION RESULTS FOR MLSE NEURAL "NEG Saas 
(TIME-VARYING GEA NEw) 


The neural network presented in this study can be thought of as an alternative to 
the Viterbi algorithm [Ref. 11] for computation of the MLSE cost function. Unlike the 
Viterbi algorithm implementation, the neural network does not require a vast amount 
of memory for storage. From the simulation results for the two transmission channel 
conditions for a channel memory of 2, we can conclude that the neural network can be 
used to estimate a transmitted sequence of binary data symbols. Comparing the esti- 
mates of the MLSE neural network and those of the direct computation of the MLSE 
cost function, we can say that The MLSE neural network does indeed perform the de- 
sired estimation. The amount of data provided by the simulations is far too little to make 
anv final conclusions concerning the performance of the MLSE neural network. How- 
ever, the results are promising and indicate that the neural network may be an attractive 


alternative for implementation of MLSE for binary communications systems. [Ref. 3] 
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VI. CONCLUSION 


A. SUMMARY OF RESULTS : 

In this study, we have first made an introduction to the field of artificial neural 
networks. Then, we described the use of some neural networksin pattern recognition and 
classification using binary pattern elements. A computer program from an algorithmic 
approach for each one of these networks was constructed and used to simulate the op- 
eration of the net for different cases of input pattern. 

The Hopfield network was the first net we worked on. A simulation program 
imlementing the operation of this net as a content addressable memory for random input 
patterns was made. As a supervised network, the Hopfield net is only iterating between 
an input pattern and the ones that the teacher has already stored in 1ts memory, showing 
that this net is a non-learning one. This net was simulated by presenting noise-corrupted 
or perfect input patterns. The response of the Hopfield net to each one of these input 
patterns was provided to show the iterations taken by the net to recognize and classify 
even noise-corrupted input patterns. By recognition and classification, we mean the net 
converges to one of the M stored patterns that best matches the input pattern, as long 
as the original pattern was stored in the net’s memory prior to its use, otherwise a “no 
match” will occur. 

However, the number of stored patterns (M) is a limitation to the proper operation 
of the net as a classifier because of the convergence condition demonstrated by Hopfield, 
which states that the net will converge with high probability if 4f < 0.15N, where N is 
the number of elements or bits in each pattern. These bits are taking on +1 and —] 
values, for the +1 and —1 states, respectively. 

The Hamming network is a classifier that calculates the Hamming distance to the 
exemplar of each stored class and select that class with the minimum Hamming distance 
to the specified input pattern. The Hamming distance is the number of bits in the input 
which do not match the corresponding exemplar bits. As a supervised network, we have 
first stored 10 exemplar patterns (M=10) in its memory prior to its simulation. Simu- 
lating the operation of this network on the computer, we have seen that it effectively 
converges to the correct class for each input pattern. Even presenting noise corrupted 
input patterns, the net correctly converges to the correct class, as long as the original 


pattern was stored in its memory prior to the simulation, otherwise a “no match” will 


TS 


occur. By convergence of the net, we mean the output nodes of the upper subnet (see 
Figure $) stop changing in time and only the output node corresponding to that 
exemplar class which best matches the input pattern, 1s a positive nonzero value. While, 
all the other output nodes Were driven to zero by inhibition. In practice, the net will 
converge and find the correct class when each weight w,,, connection weight from input 
1 to node j in the lower subnet (see Figure 8), 1s set to + plus a small random 
component. Like the Hopfield net, the elements of the patterns used in these simulations 
were taking on +1 and —1 values for the +1 and —!1 states, respectively. 

As a Self-organizing (a non-supervised) neural net, the Carpenter / Grossberg net 
self-organizes and self-stabilizes its recognition codes in response to arbitrary sequences 
of binary input patterns. In its learning process, the net uses a threshold level called the 
vigilance value which determines how good the learned categories will be. If vigilance 
value increases due to an environmental disconfirmation, then the net automatically 
searches for and learns the best recognition categories. The Carpenter / Grossberg net 
is Well known as an ART system. described in Appendix C, which forms clusters and 1s 
trained without supervision. This net can learn from input patterns and later differentiate 
between new and learned patterns. If the new and unknown pattern is classified as 
previousely learned pattern at a certain level of vigilance, it will be ignored, but if it is 
not, 1t will be added cs a new learned pattern. This process is repeated as long as the net 
is learning. The number of learned patterns thus grows with time and depends stronglv 
on the level of vigilance used to compare input to the already stored ones. The results 
of simulating this network showed clearly the importance of the vigilance threshold. The 
first simulation was done with a vigilance value of 0.9, which means that an input pat- 
tern must be very close to a stored exemplar to be considered sinular. The result was 4 
patterns learned out of 6 input patterns because the net has taken two input patterns 
as an already learned one. Next, we have done the same simulation but this time with a 
vigilance value of 0.7. The results were 2 patterns learned out of 6 input patterns pre- 
sented. So higher vigilance threshold can make the net to consider two patterns which 
are most similar, to be different and lower threshold can make the net to consider two 
different patterns as similar. Thus the vigilance threshold, used in the learning process 
of this network, is the dominant factor in the operation of this net, which behavior de- 
pends strongly on it. A proportional adaptation of the vigilance level to the existing 
noise in the channel can make the net to perform perfectly during training and testing. 
The elements of the patterns used to simulate this net take on, contrary to the Hopfield 


and Hamming nets, the +1 and 0 values for the + 1 and —I states, respectively. 
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On the other hand, bandwidth-efficient data transmission over telephone and radio 
channels is significantly improved by the use of adaptive equalization to compensate for 
the time dispersion introduced by the channel. From the work done on adaptive re- 
ceivers, a class of nonlinear receivers referred to as maximum-likelihood sequence esti- 
mation receivers have emerged as front-runners with respect to error rate performance. 
However, the high degree of computational complexity of the optimal maximum- 
likelihood receivers has prohibited their use in many applications. It was shown that 
neural networks can be used to implement the MLSE and that these networks offer an 
attractive alternative for implementation. After mapping the MLSE onto a neural net- 
work, we have done some simulations on this network for stationary and time-varving 
channels. The results, even though they are not based on enough data to draw definitive 
conclusions. showed that the neural network may be an attractive alternative for imple- 


mentation of the {LSE for binary communications. 


B. NEURAL NETWORK TASKS : 

The field of neural networks include manv different models designed to address a 
Wide range of problems in the primary application areas of speech, vision and robotics. 
Most researchers focus on neural networks that perform those seven major tasks illus- 


trated graphically in Figure 41. These tasks include : [Ref. 1] 


SENSORY OATA PREPROCESSING 
(Vision. Speech} 


CLASSIFICATION 
“CAT” MOTION, 
COLOR. 
OEPTH 


SELF-ORGANIZATION/ NONLINEAR MAPPING 


CATEGORY FORMATION 
Xy a, 
Xy a, 
ROBOTIC CONTROL 


ASSOCIATIVE MEMORIES MULTI-SENSE AUTOMATA 


Sy = - 





Figure 41. Seven Tasks that Neural Networks Can Perform [Ref. 1] 
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Pattern classification: Classifiers are trained with supervision using labeled training 
data to partition input patterns into a pre-specified number of groups or classes. 
These could represent different objects for a visual image classifier. Inputs to a 
classifier may be binary as we have seen for the Hopfield and Hamming nets or 
continuous-valued. 


Self-organization or Clustering: Self-organizing networks. like the Carpenter / 
Grossberg net, partition input examples into groups or clusters using unlabeled 
training data. This tvpe of clustering or vector quantization is an efficient technique 
for reducing information that must be processed at higher levels with little loss in 
performance. It also makes good use of the large amount of unlabeled training data 
that is typically available in speech and vision problems. 


Associative memory (storage and access): An associative, or content-addressable 
memory provides a complete memory item from a kev consisting of a partial or 
corrupted version of the memory. For example, it might return a complete article 
citation from onlv the author’s name or a complete image of a face from only the 
bottom half. 


Sensory Data Processing (vision and speech): An enormous amount of realtime 
preprocessing is performed in the peripheral sensory vision and hearing centers. 
Neural networks can perform this function in real time using massive parallelism. 


Computational Problems: Custom neural network architectures can be designed to 
solve specific computation problems, such as the traveling salesman problem and 
other constrained optimization problems, using nonlinear analog computation. 


Nonlinear Mapping: Many neural networks can map a vector of analog inputs into 
an output vector using a nonlinear mapping function which can be learned from 
training data. These types of mappings are useful in many areas, including robot 
control and nonlinear signal processing. 


Multi-sensor Automata: A number of complex, multi-module neural network 
automata have been built with visual input and a robot arm to manipulate objects 
in an environment. These automata demonstrate how an eve or camera can learn 
to scan a scene using Sself-supervision, how control of a multi-jointed arm and hand 
can then be learned using self-supervision, and then how the eye and hand can be 
coordinated to perform simple tasks. These automata also demonstrate how inputs 
from multiple sensors can be fused to provide classification performance better 
than could be achieved with a single sensor. 


C. CONCLUSIONS : 


From the study done by DARPA [Ref. 1], we can conclude that neural networks 


offer important new computational structures. Their real strength is derived from their 


ability to self-adapt and learn. If neural networks realize their full potential, they can be 


used for machine vision, speech recognition, signal processing, robotics and other ap- 


plications. 


Neural network research has matured greatly since the perceptron of 1950s, thanks 


to the development of advanced mathematical theories and new computer tools, and also 
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to a better understanding of neurobiology. The hardware capabilities are limiting the de- 
velopment of important neural network applications. It is clear that if researchers are 
not provided with improved simulation and implementation capabilities, the field of 


neural networks will once again drift off into the wilderness. 
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APPENDIX A. PROGRAMING THE HOPFIELD NET WHEN USED AS 
A CLASSIFIER : 


Using Fortran as programing language, the previously described operation algo- 
rithm of the Hopfield net when used as a classifier was implemented with the mainframe, 
and used to run some simulations as described in the simulation paragraph of the 
Hopfield net. 


FesedesesevodesedetosedesesevesetedesesetosededesetedededetekestetecctekkhctetckdeseseskdesekkRseetehseter eRe 


weevere THESIS RESEARCH deeteye 
a HOPFIELD NET SIMULATION PROGRAM a 
vedeveds BY M. H. KHAIDAR vedodeds 
Teme Tete Te Tete Te ETS eT Te TERETE TE TOTS OTe ToT TE TOTES TOTES TE TET TOTES TE TE TETE CTE TET ESTE TTS TR TS TST TOTS TET TCT TET 
ve vlevlen' fawe ste we Je se s'e we evesic se we se we steele slewe le nlon' owe we slo Seve locestes aie afoales efectos! ‘eute we we ve 1 ele ate louis arloulouis wie ulanlouiswlaciouls w! ese we ve we ste steele 
a THIS PROGRAM WAS MADE TO IMPLEMENT THE HOPFIELD NETWORK _ 


“© OPERATION ALGORITHM WHEN THIS NETWORK IS USED AS A * 
*% CLASSIFIER. AFTER THE INPUT PATTERN IS PROCESSED AS vs 
* DISSCUSSED BEFORE AND AFTER CONVERGENCE, THE OUTPUT WILL BE > 

** COMPARED TO THE M (M = 8 IN THIS IMPLEMENTATION) EXEMPLARS * 
* TO DETERMINE IF IT MATCHES AN EXEMPLAR EXACTLY. IF IT DOES, * 
“* THE OUTPUT IS THAT CLASS WHOSE EXEMPLAR MATCHED THE OUTPUT * 
*% PATTERN. IF IT DOES NOT THEN A “NO MATCH" RESULT OCCURS. vs 


DECLARATION OF VARIABLES: us 
¥ PATT(I,S) = THE ITH ELEMENT OF THE STH STOKED EXENP iis 
" TC 1A = THE CONNECTION WEIGHT FROM NODE I TO NODE i 


ze Ua) = THE OUTPUT OF NODE J AT TIME T 

zt W(I1) AND V(I) = THE ITH ELEMENT IN THE MATRIX COLUNM * 
“ INPUT PATTERN W AND THE MATRIX COLUNM 
t OUTPUT PATTERN V v 
“ MAT(12,10), V(12,10) AND CMAT(12,10) = THE 12 BY 10 * 
ze MATRIX REPRESENTATION OF AN EXEMPLAR ** 


~ CLASS(J) = THE MATRIX COLUNM OF THE JTH STORED PATTERN* 
* N = THE NUMBER OF ELEMENT IN EACH EXEMPLAR ¥ 
* M = THE NUMBER OF STORED EXEMPLARS * 
* DIFF(120,J) = THE DIFFERENCE BETWEEN THE OUTPUT MATRIX * 
5 COLUNM AFTER CONVERGENCE AND THE JTH STE 
* PATTERN FOR CLASSIFICATION 


Dededetedsdeteled PRR ICSI BEERS ERE REEREEREREEERERRERRERERBRERERERERE 


ee fy C2 C7 C7 CI CIC II OO AIIM ANSaarIanonnoaaG 


INTEGER PATT(120,8), U(10,120), T( 120,120), V(12,10), CLASS(120) 
INTEGER S, I, J, K, COUNT, MAT(12,10) 

REAL W(120), VEC(120), DIFF(120,8), CMAT(12,10), DMAT(12,10) 
CHARACTER*1 TEMP( 12,10) 

PRINT*,' HOPFIELD NETWORK IMPLEMENTATION’ 
PRINT**,' esas! 
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Crircy C2 


Ca"¢2 Ca'¢2' co 


sedeakokscckcdescacseskescacvdededcsedescsessstesdeskdcsededescscsdesdevedkedeakskesesdcsedesksedesleveskdeseseslesteocdesdesedesedesesevevese 
¥ INITIALIZE WITH UNKNOWN INPUT PATTERN % 


seseslestcscotcskcotcdesksesesevesleseceseslcsesicdescdeskdesksccleclesedeveseskevedesesesedesededeveslevesedesdkedesevesccesledesedesevedescse 


N 120 


M= 8 
OPEN(UNIT=1, FILE='NINE1', STATUS='OLD') 
fos 1-1. N 
READ(1,*) W(I) 
WC la) = ¥C1) 
5 CONTINUE 
PRINT*,‘ ' 
PRINT*,' ' 
PRINT 1 
1 FORMAT( 'THE UNKNOWN INPUT PATTERN TO THE HOPFIELD NETWORK, ' 
&'ON THE LEFT AS IM-'/ 
&' POSED ON THE NETWORK AND ON THE RIGHT IN A MUCH CLEAR ' 
&' REPRESENTATION’ / 
&'WHERE EVERY ( * ) REPLACES A 1 AND EVERY ( . ) REPLACES A -1:') 
CALL VECMAT(W,MAT) 
PRINT*,' ' 
HO 210 T=1.12 
Bom220cs—1510 
DUAet sd) = MATCI,J) 
220 CONTINUE 
210 CONTINUE 
DO 230 I=1,12 
Hor24o Je— ilo 
IF(DMAT(I,J).EQ.1) THEN 
TEMP(I,J) = ‘*' 
ELSE 
TEMPGls3) = °° 
ENDIF 
240 CONTINUE 
230 CONTINUE 
DO 140 K = 1, 12 
Piece ore MATCK,3), J=1,10), (TEMP(K,J), J=1,10) 
140 CONTINUE 
ise FORMAT( 4X ,1013,6X,10(Al, 2X)) 
HoelO I=1, N 
Ree wie, 15)) (PAIICI,S), S=1,M) 
10 CONTINUE 
15 FORMAT(1X,815) 


KacteickicvskkkikkekKeKccKKkkKkkkkkkkdedkkdkkadedkdededsdedesdedededesdesdevedededededededesdededk sever 


ve ASSIGN CONNECTION WEIGHTS ve 
sweavdedekdekkedksdkddkddesdvdededdedesedkddescadddddescvsakdcsaldkdkedededesesesdesescdoksesdekdededeavskeskesesedededeskeveskcskse 


DO 20 J=1, N 
DO 30 I=1, N 
PEGI snOn) THEN 


T(I,J) = 0 
ELSE 

SUM = 0 

DO 35 S=1, M 


SUM = SUM + (PATT(1I,S)*PATT(J,S)) 


8] 


C2 Oia) 


C2 ¢) €2 C2 CO 


60 


20 


70 


40 


400 


90 


260 


80 


+++ 


CONTINUE 
TCI,J) = SUM 
ENDIF 
CONTINUE 
CONTINUE 


Wwiewevedesivededeevesek sek sevevedededevededededdedevevetedededevedesvedkdedcdedsvodkdededevedevedetcdkdedevedvedesedetedde 


* ITERATE UNTIL CONVERGENCE us 


tevedededeadcdedesosvvededededesevevevevedevededcdescdededokvvevesssesededesesededekekededkvadedkevedk deskdedetedsdcdededevevcskese 


DO 40 K=1, 9 
DO 50 J=1, N 
SUM2 = 0 
DO 60 I=1, N 
SUM2Z°>= SUM2>+>(1C 1.) UCK ae 


CONTINUE 
TF(SUM2Z, GT. 0) THEN 
UCK+ 1) 
ELSE 
UKE — 1 
ENDIF 
CONTINUE 
FLAG = 0 
DO 70 sii 
TECUCK. 1) ONE UGG et EN 
FLAG = 1 
ENDIF 
CONTINUE 
IFC FLAG. EQ. 0) THEN 
COUNT = K 
GOTO 400 
ENDIF 
CONTINUE 


Sededevevlevlcdevevlessescsecedevdesevevesesededcaesevdesleskesestesleskseslevescdcvlesesdeseskevdesleslesevesededescdeskddese dedevedevdesiese se 


PRINT SHAPES oe 


sedevedesdedcokvlesevcsdc clove deseveveveslonesivicviesevesevevcsicvesevedesede ceded dedeseve cesses sea desk sese dk ses vedesevess oe 


DO eC 1-1, scoun 
DO=s0) J— ean 
VEC( J) = UG 
CONTINUE 
PRINT*,' ' 
PRINT*,' ' 
PRINT 260,1 
FORMAT( 'AFTER, THE',I2,'TH ITERATION(S), THE OUTPUT OF THE ' 
"HOPFIELD NETWORK LOOKS LIKE '/ 
'THE FOLLOWING FOR THE ' 
"UNKNOWN INPUT PATTERN PRESENTED: ') 
PRINT*,' ' 
PRINT*,' ' 
CALL VECMAT( VEC ,MAT) 
CALL CHARMAT(MAT,CMAT) 
CONTINUE 


Pevededede desea delete Nee RPT TCT TE LIC LCP TELE TLE TELE RE EERERERERER REE ECEREEREERREEREEBREER 


(Sp bal ap Ne ep | 


% CLASSIFICATION * 
Nevcacakoicakcakcacdededkesesescacskedkedeakcskesesesevedesevedevedevedededkevesedeakcacskededeadcacacdedcdedcacdedcalcdededeskedededevesdedese 


PRINTS. 

PRINT*,' ' 

PRINT*, ‘CLASSIFICATION OF THE UNKNOWN INPUT PATTERN: ' 
ee en ee et t 


PRINT 2 
2  FORMAT('AT THIS POINT, 
&' CHANGE ON THE OUTPUT’ / 
& 'OF THE NETWORK AND THE PATTERN SPECIFIED BY THE OUTPUT NODES ' 
&'IS THE'/ 
&'NET''S OUTPUT. THE TASK OF THE NET NOW IS TO CLASSIFY THE ' 
&' INPUT AS AN'/ 
&' ALREADY KNOWN PATTERN OR A NO MATCH WILL OCCUR. AFTER ' 
&' CLASSIFICATION, '/ 
& ‘THE OUTPUT PATTERN OF THE HOPFIELD NET MATCHES BEST THE PATTERN ' 
&'OF') 
menz00 I=1, N 
CLASS(I) = U(COUNT,I) 
200 CONTINUE 
DO 180 S=1, M 
DO 190 I=1, N 
DIFF(I,S) = PATT(I,S) - CLASS(I) 
IF(DIFF(I,S).EQ.0) THEN 


FURTHER ITERATIONS WON''T MAKE ANY ' 


SCLASS = S 
ELSE 
GOTO 180 
ENDIF 
190 CONTINUE 
GOTO 170 


180 CONTINUE 
170 IF(SCLASS. EQ. 1)THEN 
PRINT**, 'DIGIT ZERO. ' ; 
ELSEIF(SCLASS. EQ. 2) THEN 
PRINT*, DIGIT ONE' 
ELSEIF(SCLASS. EQ. 3)THEN 
PRINT*,'DIGIT Two. ' 
ELSEIF(SCLASS. EQ. 4) THEN 
PRINT*, ‘DIGIT THREE. ' 
ELSEIF( SCLASS. EQ. 5) THEN 
PRINT*, ‘DIGIT FOUR. ' 
ELSEIF(SCLASS. EQ. 6) THEN 
PRINT*, ‘DIGIT SIX. ' 
ELSEIF( SCLASS. EQ. 7) THEN 
PRINT*, ‘BLOCK REPRESENTING THE POINT. ' 
ELSEIF(SCLASS. EQ. 8) THEN 
PRINT*, 'DIGIT NINE. ' 
ELSE 
PRINT*,'NO MATCH' 
ENDIF 
CLOSE (1) 
CLOSE (2) 
STOP 
END 
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iL ILG; 
100 


Ee, 


130 
120 


165 
160 


167 
166 


re) Cire) C2 C202 C2 C2 


Seavavdvdededcvcadkvak veseseakvsdeveakvvdevedesevedededcsesevesesiese desc dededesededescatsedevevosesedvvevesesvesesededeteakvscdcdcdcveste 


SUBROUTINE VECMAT(ARR,MAT) 
DIMENSION ARR(120), MAT( 12,10) 
m= 
DO 100 J=1, 10 
DO 11%0° 112 
K=K+1 
MAT(I,J) = ARR(K) 
CONTINUE 
CONTINUE 
RETURN 
END 


Iesedeakvavdavseakaesesedke desks seve vesesed se ak desevevesesesedk weve reves sewed sesesedesesesesdeseacaeske teskesesesescse dese te sesese 


SUBROUTINE CHARMAT( MAT, CMAT) 
DIMENSION MAT(12,10),CMAT( 12,10) 
CHARACTER*1 TEMP( 12,10) 
DO 120 1=1,12 
DO 130 J=1,10 
CMAT(I,J) = MAT(I,J) 
CONTINUE 
CONTINUE 
DG 160 [=i 
DO 165 4e—elo 
IF(CMAT(I,J). EQ. 1) THEN 
TEE Cis) eae 
ELSE 
MEMP(I J) = 2- 
ENDIF 
CONTINUE 
CONTINUE 
DOmloon 1) — see 
WRITE(6,167)(TEMP( I,J) ,J=1,10) 
FORMAT( 22X,10(A1,2X)) 
CONTINUE 
RETURN 
END 


sevevesevevescdevescvleseslesicvleseselcskscveaccleslcsicsedeslesicskotestescacseskokocseskseseuesesleskoescnesteoesescslestskscavotst ses ses 


* HERE ARE THE 8 STORED EXEMPLAR PATTERNS USED IN THIS PROGRAM * 


* FOR CONVENIENCE, I CHOOSED TO WRITE THEM IN COLUNMS WHERE 

EACH ONE CORRESPONDS TO A STORED EXEMPLAR PATTERN. THE FIRST 
* COLUNM CORRESPONDS TO THE PATTERN OF A ZERO, THE SECOND OF A 
* ONE, THE THIRD OF A TWO, THE FOURTH OF A THREE, THE FIP ines 


sf 


mis 
aa 
TAY 
* 


ate 
ee 


* A FOUR, THE Six TneGr 24 SIX, THE SEVENTH OF A POINT, THE abe 


* AND LAST OF A NINE. 


Seredevevedeatseskesteskskste koe sksledeseakskeakseotk skesakskacotakcskc kok skakseskseskcukotse ak seskatsesk sk skese ak ak ales ae ak ak sie aie ake te ae 
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ee ed ee 
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APPENDIX B. PROGRAMING THE HAMMING NET WHEN USED AS 
AN OPTIMUM CLASSIFIER : 


Using Fortran as programming language, the previously described operation algo- 
rithm of the Hamming net when used as a classifier was implemented with the 
Mainframe, and used to run some simulations as described in the simulation paragraph 


of the Hamming net. 


Sosleveveslovevodededevedesevevedevetedevesekdededeledesedeseteseveltesesesesededetedsedleddededksededesedevek Herve 
weadede THESIS RESEARCH feet 
sein HAMMING NET SIMULATION PROGRAM fetete 
“ges BY M. H. KHAIDAR even 


Sevevevedevedevedededsdedetetetidetetetedetetededcteiciviciivinggcc“diclclhdvchddcdclekck dedectcdoldckdededededs 


ORV every es ern ee ed ed ria OR FR FR IR EDR GER GR ER ER HE HR ED FR EE GS HIRED FR ERX FS FL ER ER FX FB FD seen » aces ey wae arve eee DR ER eR FL CR 4d ri erty ee eyed 


v9 duaeue 


* THIS PROGRAM WAS MADE TO IMPLEMENT THE ALGORITHM OPERATION OF* 
%* THE HAMMING NET, WHEN IT IS USED AS A CLASSIFIER, PROVIDED IN* 
* THE CHAPTER ABCUT THIS SNE: w 
VARIABLE DECLARATION : 
¥ N = NUMBER OF NODES IN EACH EXEMPLAR ie 
% M = NUMBER OF STORED EXEMPLARS * 
* PATT(I,J) = THE ITH ELEMENT OF THE JTH STORED EXEMPLARS 
# THETA = THE THRESHOLD IN EACH NCDE a 
7 WC I,J) = THE CONNECTION WEIGHT FROM INPUT I TO NODE J * 
¥ X(I) = THE ITH ELEMENT OF THE INPUT PATTERN TO THE NET ee 
* UC J,T) = THE OUTPUT OF NODE J AT TIME T * 
a EPSILON = THE VALUE OF WEIGHTS (INHIBITORY) BETWEEN * 
7 DIFFERENT OUTPUT NODES He 


Sesevededevededetodetidevetetetetelelededetetotiteieleleictedekdcloiodedededededededetedededs Feed SeRe ele le RRR A AS 


C2 G2 C2-C2 Ca-C2C) C2.) Ca Ga: C2 C2 C2 Ca) G10) G2 2 CG & 
of 


INTEGER PATIC120,10), UC1001)) eat ee 

INTEGER THETA, 1, J,79 Ky 7 ARG ero) 

REAL RESLT, EPSILON, WC 120,10), SUM, SUM2, XC12Z0) Ge ALE Gra 
REAL CHATCUZ. 10) 

CHARACTER*1 TEMP( 12,10) 

PRINT*, | HAMMING NETWORK IMPLEMENTATION’ 

PRINT*, ' sono se er ee eee 

EPSILON = 0.08 


weereverddee ~THETA( J) = N/2 = 120/2 = 60 kitriedetebietdeidetk eee 


aman 


THETA = 60 


Heevededevetedetedetecedededededededededetedodededtetesete RR RIKER REE ERRERREREEREREREREREE 


* ASSIGN CONNECTION WEIGHTS * 
Setededevededevetededetetededete teve detec tete fete Fete teh Te fet AIRRRERERRERRREREREER 


G)G7.C) ©) C2 


M = 10 
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C2/Ci CG) 2 


N = 120 
DO 10 I=1, N 
READ(*,15) (PATT(I,J), J=1, M) 
10 CONTINUE 
15 FORMAT(1X,1015) 
DO 20 I=1, N 
DO 25 J=1, M 
Creo) = PATHOL. J) /2. 0 


5 CONTINUE 

20 CONTINUE 
ServevevievicveveveseveveveteeedesetekidetekietecietehkkeietedesescieietetictlehdededledelekdeeieckckedleiehRickkdichdsede 
* INITIALIZATION WITH UNKNOWN INPUT PATTERN * 
Sedeue fe dausdeclenecoede secede cece se eC eee Cee eC ee RMS etiegeeeiciciisisienqgikiecsdclliisheedteds: 


OPEN(UNIT=1, FILE='INPUT', STATUS='OLD') 
perso I=], N 
READ(1,*) X(I) 
30 CONTINUE 
PRINT*,' ' 
PRINT*,' ' 
PRINT 1 
1 FORMAT('THE UNKNOWN INPUT PATTERN TO THE HAMMING NETWORK, ON THE' 
&' LEFT AS IM-'/ 
&' POSED ON THE NETWORK AND ON THE RIGHT IN A MUCH CLEAR’ 
&' REPRESENTATION’ / 
&' WHERE EVERY ( * ) REPLACES A 1 AND EVERY ( . ) REPLACES A -1:') 
CALL VECMAT(X,CMAT) 
CALL CHARMAT( MAT ,CMAT, TEMP) 
PRINT*,' ' 
PRINT*,' ' 
DO 140 I =1, 12 
Horio J=15 10 
MAP(I,J) = INTCCMAT(I,J)) 
141 CONTINUE 
140 CONTINUE 
momt4? IT =1, 12 
WRITE(6,145)(MAP(I,J), J=1, 10), (TEMP(I,J), J=1, 10) 
142 CONTINUE 
145 FORMAT(4X,1013,6X,10(Al1,2X)) 
DO 40 J=1, M 
SUM — 10 
NC) Si) a IN 
SUM = SUM + W(1I,J)*X(I) 
50 CONTINUE 
SUM1 = SUM + THETA 
IF(SUM1.GT.0) THEN 
UCJ,1) = SUM1 
ELSE 
UCJ,1) = 0 
ENDIF 
40 CONTINUE 


vevedvedededededestesedevedieleteieiieesdesckkkiciciodenkhKntkkekRckddlkiicckEsicsk 


* ITERATE UNTIL CONVERGENCE * 
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ep Fl pp | <p I op | gi 


Ci, C2 Ca-G? Case) 


80 


70 
60 


90 
95 


3 


Devevedevevesvadededededlede Hele vecesevevedesesdecevesedevedesvacdedededesesevcukcvedesesdeuke sedkcovsesdevesdcdkcsvesccvskcscseacseveseseste 


DO 60 T=1, 10 
DO 70 J=1, M 
SUM2 = 0 
DO 80 K=1, M 
IF(K.NE. J) THEN 
SUM2 = SUM2 + U(K,T) 
ENDIF 
CONTINUE 
RESLT = U(J,T) - SUM2*EPSILON 
IF(RESLT. GT. 0) THEN 
U(J,T+1) = RESET 
ELSE 
U(J,T+1) = 0 
ENDIF 
CONTINUE 
CONTINUE 


Sedeovcesecesedededeskkavsvsededcsevesestevoskesesvsesescskcskesdkedesesesese sealed avseseslest seskcsicseskeseskeskcskesksesesesesesesesese oe 
7 THE OUTPUT OF THE HAMMING NETWORK * 


aevedesesedeskcakcovslcslcsesesesescvleutslevesleskeacaevevesleseslccc oksleseslescsaicckevesk veveskeseseseveseseseseskcsdkeskcseseveseseskeseseaksese 


PRINT*,'° 

PRINT*,' ' 

PRINT 2 

FORMAT('THE OUTPUT OF THE HAMMING NETWORK, WHERE EACH COLUNM ' 


& REPRESENTS THE / | 

&' OUTPUT NODE VALUES FOR THE CORRESPONDING CLASSES AT A CERTAIN ' 
&'NUMBER '/ 

&'OF ITERATIONS: ‘) 


PRINT*,° 
PRINT*,' ' 


Sercakvslcskslcceskcscoestsseseseseckscseslestcesesicacseatslestsesiescsksesescacakcslesteskeakcskskeseskskeskskscseskocskseseskseslesesk seater 
ONLY TEN ITERATIONS ARE SUFFICIENT TO THE NET TO CONVERGE TO~* 


¥ THE RIGHT ANSWER FOR OUR SIMULATIONS * 
ee oi oor fir ii rk a Dirtie k oiriccr beet bark ba bie iicoir birtorise (oi iirtacird acta fart vcore sorter 
PRINT*,'NUMB. OF ITERATIONS= 1',' 2 Bias 4 o- 
t Be rhe Bt a! 70. 
PRINT*,' | 
DOS 0 =i. 

Q=J-i1 

WRITE(*,95) Q,CUQJCT), Totes 
CONDINUE 


FORMAT(1X,'FOR CLASS’ ,I2,': ' ,4X,1015) 

PRINT*,' ' 

PRINT*,' ' 

PRINT, ‘CLASSIFICATION OF THE UNKNOWN INPUT PATTERN: | 

PRINT % ,° === 

PRINT 3 

FORMAT( 'THEN, THE DISTURBED UNKNOWN INPUT TO THE HAMMING NETWORK ‘ / 


&' AFTER CONVERGENCE CORRESPONDS TO THE PATTERN STORED OF THE’ ) 


IF(U( 1,10). GT. 0)THEN 
PRINT, ‘CLASS ZERO. ' 
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pe i 


a Ae 1s | 


LO 
100 


160 
ee 


180 
170 


ELSEIF(U( 2,10). GT. 0)THEN 
PRINT*,'CLASS ONE. ' 
ELSEIF(U(3,10). GT. 0)THEN 
PRINT*, ‘CLASS TWO. ' 
ELSEIF(U(4, 10). GT. 0)THEN 
PRINT*,'CLASS THREE. ' 
ELSEIF(U(5,10). GT. 0) THEN 
PRINT*, ‘CLASS FOUR. ' 
ELSEIF(U(6,10). GT. 0) THEN 
PRINT*,'CLASS FIVE. ' 
ELSEIF(U( 7,10). GT. 0) THEN 
PRINT*, 'CLASS SIX. ' 
ELSEIF(U(8,10). GT. 0) THEN 
PRINT*,'CLASS SEVEN. ' 
ELSEIF(U(9,10). GT. 0)THEN 
PRINT*, ‘CLASS EIGHT. ' 
ELSEIF(U( 10,10). GT. 0) THEN 
PRINT*, ‘CLASS NINE. ' 
ENDIF 
STOP 
END 


Feveseseskcdeskcvdedesoseaesededesk deseaevedesedeaeskecde desk seals vest ak sede ae ae ae ak ae seve ae teak veak se aeakeaacakvake sessed seve sess se 


SUBROUTINE VECMAT(ARR,CMAT) 
DIMENSION ARR(120), CMAT(12,10) 
K = 0 
DO 100 J=1, 10 

DO 110 I=1,12 


K=K+1 
CMAT(I,J) = ARR(K) 
CONTINUE 
CONTINUE 
RETURN 
END 


Jeveseavaesecedesesesedededededesesedededededesevevesevevescskesedesevesvseskeakseveskedesesdevesesdedeseskskesesesdesde se desevese cece ae 


SUBROUTINE CHARMAT( MAT, CMAT, TEMP) 
DIMENSION MAT(12,10), CMAT( 12,10) 
CHARACTER*1 TEMP( 12,10) 
DO 150 J=1, 10 
BO 160 I=1, 12 
MAT(I,J) = CMAT(I,J) 
CONTINUE 
CONTINUE 
DO 170 I =1, 12 
DO 180 J=1, 10 
IF(MAT(I,J).EQ. 1) THEN 
TEMP(I,J) = '*! 


ELSE 
TEMP(I,J) = °.' 
ENDIF 
CONTINUE 
CONTINUE 
RETURN 
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END 


dertevedederevedevevevevede deve weve ve ve veve deve ws ved te ve ve sose Te vets He te te dese deve ge ve seve Fe ve Fe Fe Ve se Fe te Fe Fe Te ee Te Fete He ves 
* HERE ARE THE 10 STORED EXEMPLARS USED IN THE IMPEEMENTATICN@ Ss 
* OF HAMMING NET. FOR CONVENIENCE, I CHOOSED TO WRITE THEM IN] * 
* A MATRIX OF 10 COLUNMS AND 120 ROWS, WHERE EVERY COLUNM % 
* CORRESPONDS TO AN EXEMPLAR OF 120 ELEMENTS REPRESENTING THE * 
* 12 BY 10 REPRESENTATION OF THE PATTERN AS SHOWN IN THE STUDY * 
* OF THE HAMMING NET. THE FIRST VECTOR COLUNM CORRESPONDS TO % 
* THE PATTERN OF DIGIT O, THE SECOND OF 1, THE THIRDSGE 2. ae 
* FOURTH OF 3, THE FIFTH OF 4, THE SIXTH OF 5, THE SEVENTH SG 
* 6, THE EIGHT OF 7, THE NINTH OF 8 ANDVUHE SAS Ore ¥ 


Jeakcevicsestesleseseseccleseslesicnke sk ckckclcsiesioseskaccseseseslcsesleskslesesesleveseicdesesesesescvesioveseslevesesest sk seseslevevevevedess 


=i 1 =1 =all =} i =I = =a 
=i 1 al =u ali 1 Sal = =1 
= = =a =a = JL = =I ae 
=I =] =1 = =i 1 «ll =a a 
=i al al =i = 1 = a1 =] 
=r if oul Sal =i 1 =1 al «i 
= 1 S| =i al 1 =al = = 
= 1 = =1 a 1 a1 =1 =1 
il 1 mel =a = 1 eal all oul 
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APPENDIX C. ART AND OPERATION OF THE CARPENTER / 
GROSSBERG NET : 


The following is a description of the ART net operation according to Carpenter / 


Grossberg [Ref. 12]. A cvcle that traces the real ttme dynamics of ART network in re- 


sponse to arbitrary sequences of binarv input patterns 1s depicted in Figure 42. 





Figure 42. The ART net search for a correct F, code. [Ref. 12] 


In Figure 42a, an unknown input pattern I is presented to the net. Pattern I is then 
transformed into a pattern X of activation across the nodes.In other words, the input 


pattern I generates a short term memory (STM) activity pattern X across a field of 


22 


feature detectors fF, . Grossberg sees short term memorv (STM) as a wav of keeping 
patterns active after the original input pattern has vanished. A short term memory is a 
persistent activity pattern in a set of neurons, maintained by nonlinear feedback system. 

The input pattern I also activates an orienting subsystem A, but pattern X at F, in- 
hibits A before it can generate an output signal. On the other hand, the pattern X of 
STM activities across F,, elicits an output pattern S of output signals from F,. When a 
signal from a node in F, 1s carried along a pathway (the bottom-up adaptive filter) to 
F,, the signal is multiphed or gated by the pathway’s long term memory (LTM) trace. 
The LTM-gated signal (1.e., signal times LTM trace), not the signal alone, reaches the 
target node. Each target node sums up of all of its LTM-gated signals, which results in 
pattern S generating a pattern T of LTM-gated and summed input signals to F, as shown 
in Figure 42a. The transformation from S to T is called an adaptive filter. The input 
pattern T to F, is quickly transformed by interactions among the nodes of F, . The re- 
sulting pattern of activation across F, is a new pattern Y. This new pattern, rather than 
the input pattern T, is stored in STM bv F,. As soon as the bottom-up STM transfor- 
mation Y — }' is completed, the STM activities Y in F, elicit a top-down excitatory signal 
pattern U back to F, (Figure 42b). Only sufficiently large STM activities in Y elicit 
signals in U along the feedback pathways F, > F,. As before, the top-down signals U are 
also gated bv LTM traces and the LT M-gated signals are summed at F, nodes. Then, the 
pattern U of output signals from F, generates a pattern V of LTM-gated and summed 
input signals to F,. The transformation from U to V is thus also an adaptive filter. The 
pattern V is called a top-down template. or learned expectation. 

Two sources of input now perturb F,, the bottom-up input pattern I which generated 
the original activitv pattern X and the top-down template pattern V that resulted from 
activating X. The amount by which activity in X 1s attenuated to generate A” depends 
upon how much of the input pattern I is encoded within the template pattern V. In 
particular, F, acts to match V against I. Now, we will discuss how a match or mismatch 
of I and V at F, regulates the course of learning in response to the pattern I. 

When a mismatch attenuates STM activity across F,, the total size of the inhibitory 
signal from F, to A is also attenuated. If the attenuation is sufficiently great, inhibition 
from F, to A can no longer prevent the arousal source A from firing. Figure 42c shows 
how disinhibition of A can result in the release of an arousal burst to F, which equally, 
or nonspecifically, induces selective and enduring inhibition of active population of F,. 

In Figure 42c, inhibition of Y leads to removal of the top-down template V, and 


thereby terminates the mismatch between I and V. Input pattern I can thus reinstate the 
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Original activity pattern X across F,, which again generates the output pattern S from 
F, and the input pattern T to /,. Due to the enduring inhibition at F,, the input pattern 
T can no longer activate the original pattern Y at F,. A new pattern Y’ is thus generated 
at F, by I (Figure 42d). 

The new activity pattern }” reads out a new top-down template pattern V’. If a 
mismatch again occurs at F,, the orienting subsystem is again engaged, thereby leading 
to another arousal-mediated reset of STM at Fj. In this way, a rapid series of STM 
matching and reset events may occur. Such an STM matching and reset series controls 
the system’s search of LIM by sequentially engaging the novelty-sensitive orienting 
subsystem. Although STM is reset sequentially in time via this mismatch mediated, 
self-terminating LIM search process, the mechanisms which control the LTM search 
are all parallel network interactions rather than serial algorithms. Such a parallel search 
scheme continuously adjusts itself to the system’s evolving LTM codes. In general, the 
Spatial configuration of LTM codes depends upon both the system’s initial configuration 
and its unique learning history, and hence cannot be predicted a priori by a pre-wired 
search algorithm. Instead, the mismatch-mediated engagement of the orienting subsys- 
tem realizes the type of self-adjusting search. 

The mismatch-mediated search of LTM ends when an STM pattern across F, 
reads-out a top-down template (V) which matches I to the degree of accuracy required 
by the level of attentional vigilance, or which has not vet undergone anv prior learning. 
In this case, a new recognition category is then established and a new bottom-up code 


and new top-down template are learned [Ref. 6]. 


8) 


APPENDIX D. PROGRAMING THE CARPENTER / GROSSBERG NET 


Using Fortran as programing language, the previousely described clustering algo- 
rithm of the Carpenter / Grossberg net was implemented with the Mainframe, and used 


to run some simulations as described in the simulation paragraph of the net. 


CG Tavdevedevevede sete Tove Tove Te Fe Fe Ve He Tete Ve Fe Fete Fe TENT FETE Ke FETC REN TC NITE RETR REE KERR CR 
G tevevede THESIS RESEARCH deveveve 
C Feve ded CARPENTER / GROSSBERG NET SIMULATION PROGRAM WAKE 
C wcacde7C BY M. H. KHAIDAR Jevededk 
G Jevededededevesesvede seve teve de Teds Teves Te Fete FeNe TeV Tee TC Te FoF Te TCT He ATC TCI TCI IIIA ICICI I TS Te te Ie eee 
C 

C 

e Fedetedodededevevededede deve de dete ve de tedeve deve deve Fevete eve WIKI I Te eve Fe Fe Fe Te He Fede NeTek Toke Ke Tevet Ke eK eM 
C * THIS PROGRAM WAS MADE TO IMPLEMENT THE ALGORITHM OPERATTION * 
C * OF THE CARPENTER / GROSSBERG NET, WHEN IT IS USED AS A CLASS-* 
C * IFIER, PROVIDED IN THE CHAPTER FOR THIS NETWORK. ¥ 
C * VARIABLE DECLARATION : * 
C ¥ WC(I,J) = THE TOP DOWN CONNECTION WEIGHT BETWEEN INPUT ¥ 
C * NODE I AND OUTPUT NODE J * 
C ¥ COUNT = THE NUMBER OF PATTERNS STORED IN THE MEMORY OF * 
C * THE NET AT A CERTAIN TIME T CTHIS NUMBER IS VAR=2 
C % PANT IN GME) * 
C % BCI,J) = THE BOTTOM UP CONNECTION WEIGHT BETWEEN INPUT * 
C ¥ NODE I AND OUTPUT NODE J ¥ 
C ¥ RO = THE VIGILANCE THRESHOLD WHICH INDICATES HOW CLO-* 
C %* SE AN INPUT MUST BE TO A STORED EXEMPEAR TO ¥ 
C * MATCH % 
C * PATT(1,J)= THE ITH ELEMENT OF THE JIH STOREDSE ieee ¥ 
C % X(I) = THE VECTOR REPRESENTATION OF THE INPUT PATTERN * 
C 7% JMAX = THIS VARIABLE INDICATES THE CLASS THAT BEST % 
C we MATCHES THE INPUT PATTERN * 
C ¥ UCJ,T) = THE OUTPUT OF OUTPUT NODE J fiat eae ¥ 
C Gr ke bar tire baer ar pica hic ac tarior oir barter ok trie oirtar iitachir or ferrari ttacarbkscrcorior odode torre Ge 


INTEGER W(64,10), ANS, COUNT, JMAX, J, I, K, PATT(64,10) 
INTEGER T, AMAT(8,8), MATRIX(8,8), TRUE, TIME 

REAL B(64,10), SUM, SUM1, SUM2, SUM3, SUM4, RO, SUM5, SUM6 
REAL EPSILON, RATIO, Y(64), RESLT, CMAT(8,8) 

REAL ARR(64), X(64), PMAT(8,8), BMAT(8,8), U(10,11) 





C 
C Terededededewe Fede te seve ve Fevede vee deve Fete Fede de Fete Fe Te Fe vee Te Fe TT CT CRE REAR I RT EERE RR EEE 
C * INITIALIZATION =f 
G Teese dererededede ve deve ve sede de deve dese de vese Tove Fe sete Tete Tete Ve Ve Te Te Tee Te Te Tee TEN Te IN KER RR KR 
C 
PRINT*, ' CARPENTER / GROSSBERG NETWORK IMPLEMENTATION’ 
PRINT*, ' SS ee eee | 


DO 10 I=1, 64 


DO 20 J=1, 10 
Wel, ). = 1 
BCL, J): = 10. 125 
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LS Je og a a I a agp 


20 CONTINUE 
10 CONTINUE 
COUNT = 1 


seslesleskeslesesicsicskcslievesicsicslcsesicveccevevescsevedededevesesescvevevesedescsdedededevesededevcscskeckvskcscckskevcscsdestcdcakesleslescsvlesics'c 


¥ APPLY NEW INPUT * 
teovekolbokseseceslcccacslesesedededevededcvedededesvetedededededodededodcdedesedeseveds dedevedevedtevceseskdesesesdesededcsededevesede 


OPEN(UNIT=1, FILE='LETTER', STATUS='OLD' ) 
DO 25 I=1, 64 
READ( 1,26) PATT(I,1) 
N@ipe = PATICI,1) 
25 CONTINUE 
26 FORMAT(1X,15) 
PRINT*,' | 
PRINT*,' ' 
PRINT*,'THE FIRST INPUT PATTERN TO CARPENTER / GROSSBERG NET.:' 
CALL VECMAT(X,CMAT) 
PRINT*,' ' 
PRINT*,' ' 
DO 240 I=1, 8 
DO 250 J=1, 8 
AMAT(1,J) = INT(CMAT(I,J)) 
250 CONTINUE 
240 CONTINUE 
CALL CHARMAT( AMAT, PMAT) 
TRUE = 0 
JMAX = 1 
TIME = 0 
GOTO 800 
400 IF(TRUE.EQ.4) GOTO 600 
PRINT*,' ' 
PRINT*,' ' 
PRINT*, ‘PLEASE, ENTER YOUR CHOICE: ' 


PRINT*, ' (1) A NEW INPUT PATTERN' 
PRINT*, 

PRINT*, ' (2) STOP’ 

READ* , ANS 


IF( ANS. EQ. 1) THEN 
TRUE = TRUE + 1 
IF( TRUE. EQ. 1) THEN 
GOTO 1 
ELSEIF( TRUE. EQ. 2) THEN 
COTO R? 
ELSEIF( TRUE. EQ. 3) THEN 
GOTO 3 
ELSE 
GOTO 4 
ENDIF 
ELSE 
GOTO 600 
ENDIF 
i, OPEN(UNIT=2, FILE='E', STATUS='OLD' ) 
beeso J=1, 8 
REABC2, 35) (MATRIX(I,J), J=1,8) 
30 CONTINUE 
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C) Cav? G23 


SD 


45 
44 


50 


40 


900 


80 


FORMAT( 1X, 815) 
GOTO 7 
OPEN(UNIT=3, FILE='F', STATUS='OLD' ) 
DOecie = eee 

READ( 3,36) (MATRIX(I,J), J=1,8) 
CONTINUE 
FORMAT( 1X, 815) 
GOTO 7 
OPEN(UNIT=4, FILE='FPRIME', STATUS='OLD') 
DO 3251=1ee 

READ(4 37) (MATRIX@Id pme= leon 
CONTINUE 
FORMAT( 1X, 815) 
GOTO 7 
OPEN(UNIT=5, FILE='FDPRIME’', STATUS='OLD') 
DO 33m1=ie 8 

READ(5,38) (MATRIX(1.J)jee= ie) 
CONTINUE 
FORMAT( 1X,815) 
DO G2 T=] eS 

DO-45 d—ee 

BMAT(I,J) = REAL(MATRIX(I,J)) 

CONTINUE 
CONTINUE 
CALL MATVEC( BMAT,X) 


kivdedededelodvoetedetekkckkkickkkvkvckkkakdescdedkdedededkedkedetvdvakedeskvakcdedkvdkvavabdecesdvdcdededvdedededcdvskde 


* COMPUTE MATCHING SCORES % 


Jertededededcocdededededededededededevedededevesededesdkvvededevesevededcdedesvlevecsevedeacakseskvdkekveskskvededcsdesdededededosededede 


DO 40 J=1, COUNT 
SUM = 0 
DO 50 I=1, 64 
SUM = SUM + B(I,J)*X(I) 
CONTINUE 
U(J,1) = SUM 
CONTINUE 


Seavdvdevedededesdedevededededk devede desea svestedededesededededededekedkededkededededkvdeseseskeskedededescdedesedcsdededededssksed sed 
* SELECT BEST MATCHING EXEMPLAR “ 


Yedvavdesdedokueskeakvdkvdedkbdkvakeslcdedevevecdkcdeslesescdesedodksakesodkdkcdedescvedvaedededcsesdvadkeuedeaksvsvdvaededvdeveveveseskeseve dead 


EPSILON = 0.08 
DO 60 T=1, 10 
DO 70 J=1, COUNT 
SUM1 = 0 
DO 80 K=1, COUNT 
IF (K.NE.J) THEN 
SUM1 = SUM1 + U(K,T) 
ENDIF 
CONTINUE 
RESLT = U(J,T) - SUM1*EPSILON 
IF(RESLT. GT. 0) THEN 
U(J,T+1) = RESLT 
ELSE 
U(J,T+1) = 0 


100 


j 
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70 
60 


100 


120 


300 


47 


46 


ENDIF 
CONTINUE 
CONTINUE 
DO 90 J=1, COUNT 
PEGUCI. 2). Glo) THEN 
JMAX = J 
ENDIF 
CONTINUE 


Sererravaesdcatcacsoccevescacsedesesedevevescscskeskedesesedesdkcaktdeslesodcsdedeveseseatvededkcakesesedeslcsevedesestcslesdeskcsesedesdesdevdeved 


* VIGILANCE TEST ¥ 
skocacsedescacsesdcakcdeskdedesesesededesevevedkacakevesesledevovedesevedeslesdevescskedeskesdesdescvllesesedevededesedededesedevevesededess 


RO = 0.7 
SUM2 = 0 
DO 100 I=1,64 

SUM2 = SUM2 + X(I) 
CONTINUE 
SUM3 = 0 
DO 120 I=1, 64 

SUM3 = SUM3 + W(I,JMAX)*X(I) 
CONTINUE 
RATIO = SUM3 / SUM2 
IF(RATIO. GT. RO) THEN 


GOTO 200 
ELSE 
GOTO 300 
ENDIF 
OSES GCSE Si Te ol ea crane geil Salter air ir here Re ORT oir oar arn Oi Gok are bar oi Cie i oer Gack oi or orci one ibe or oe bar 
* DISABLE BEST MATCHING EXAMPLAR * 


river tei iwiwiwiwiwiwiwiwiwiwiwiwiwiwi@liwieiwiwiwiwiwiwiwiwi«iwiwiwiwiwiwiwiwiwiwiwiwiwiwiwiwi wi wiwiwliwiwiwiwiwiwiwiwiwiwiwiwiwiwt 


IF(TIME. NE. COUNT) THEN 
DO 46 J=1, COUNT 
TIME = TIME + 1 
IF(U(J,1).NE.0) THEN 
IF(J. NE. JMAX) THEN 
SUM5 = 0 
DO 47 I=1, 64 
SUM5 = SUM5 + B(I,J)*X(I) 


CONTINUE 
U(J,1) = SUMS 
ELSE 
U(J,1) = 0 
ENDIF 
ENDIF 
CONTINUE 
GOTO 900 


ENDIF 

PRINT*, ‘BECAUSE, THE RATIO IS LESS THAN THE VIGILANCE THRESHOLD’ 
PRINT*,'THE INPUT PATTERN IS CONSIDERED TO BE DIFFERENT FROM ' 
PRINT*,' ANY EXAMPLAR PATTERN STORED. THIS INPUT PATTERN IS' 
PRINT*,' THEN STORED WITH THE OTHERS AS A NEW EXAMPLAR. ' 

CALL VECMAT(X,CMAT) 

PRINT*,* | 
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GO) &) Ga) 


270 
260 


W650, 


140 


145 


48 


49 


200 


S10) 


PS? 
i> 


PRINT*,'THE UNKNOWN INPUT PATTERN TO CARPENTER/GROSSBERG NET.: ' 
PRINT*,' | 
Doe 260 Iie 

DOs 27.0, J—1saee 

AMAT(I3J0e=— IND(CHATGL =a 

CONTINUE 
CONTINUE 
CALL CHARMAT( AMAT, PMAT) 


Yededededoedededeved sede dedeesesesekededetok eisselekiciodekekkevecdcicntvlioldddechsedetdetdeleleletokeketete 


* ADD THE NEW INPUT PATTERN TO THE MEMORY OF THE NET * 


2 2 = ete onana 2 = ate lewesafouteuts eelewsutee'salonteulauto leuteatevleutoute ewen'a ae eae » 
TevededesededesedeseT ied eek Rested tote he teste eve tetetetesedtedede tote tededededetededs tetetehetededetevecee tesevevevete dee 


COUNT = COUNT + 1 
OPEN(UNIT=1, FILE='LETTER' , STATUS='OLD') 
DO 130 I=1, 64 
PATECI , COUNT) = <a) 
CONTINUE 
REWIND 1 
DO 140 I=1, 64 
WRITE(1,145) (PATTCI Jee COUN 
CONTINUE 
FORMAT( 1X, 1015) 
SUM6 = 0 
J = COUNT 
DO 48 I=1, 64 
SUM6 = SUM6 + W(I,J)*X(I) 


CONTINUE 

DO 49 I=1, 64 
B(I,J) = (W(I,J)*X(I)) / (0.5 + SUM6) 
W(I,J) = W(I,J)*X(I) 

CONTINUE 

TIME = 0 

GOTO 400 


Fesevecedede decesese de ee Te Tete eve Seve tess Lede de Se de Se Rede RCT RR CRATER REET TRE RRE TCE 


* ADAPT BEST MATCHING EXAMPLAR * 


whe wlonte wouter an anlenisntsnienionioctoniouteelen'se'e wien sats ule ula ula als wlan! svlonton sn! snion' sats alavlesulou'satantes aie ula el aulescieniautsclociautsate Je ate ule ale ule gfe cle wlentseleafeats 
CRANES EN EN ALES TC EN OS ED ELAN EC ER ONL EN ES EN ENED EC ENE AN EN EN EC ES ENTS ELEC EN EU EV ONES EC EN ES EN EN ES TREN ECC EC ER ER TN ON OREN ES EN EN GN EVEN GN GD 


PRINT*,' BECAUSE, THE RATIO IS GREATER THAN THE VIGILANCE' 
PRINT*, ‘THRESHOLD, THE INPUT PATTERN IS CONSIDERED ' 
PRINT*,'TO MATCH A STORED PATTERN WHICH IS UPDATED BY' 
PRINT*, 'PERFORMING A LOGICAL ''AND'' OPERATION BETWEEN ' 
PRINT*,'ITS BITS AND THOSE OF THE INPUT PATTERN, AND' 
PRINT*,'THE NEW UPDATED PATTERN WILL LOOK LIKE: ' 
DO 150 I[=ieees 

Y(I1) = PATT(I, JMAX)*X(I) 
CONTINUE 
CALL VECMAT(Y,CMAT) 
PRINT*,'° ' 
DO 155 I=1, 8 

DO 157 J=1, 8 

AMAT(I,J) = INT(CMAT(I,J)) 

CONTINUE 

CONTINUE 
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ESS 


i ——- 


CaCO? CIO) 


aqaq 


aqana 


160 


7 0 
175 
800 


180 


a0 


600 


ZOO 
220 


320 


CALL CHARMAT( AMAT, PMAT ) 


sesicsksicsksescseckecescslesicsicsacsesesccvslskveveacvedcakakakaedeseveseseskcakededvdeskedescakvckvcvdedesevleveveveveseve ded ve deve ve uke ake 


* THE UPDATED PATTERN IS PUT BACK INTO THE MEMORY OF THE NET * 


Sesesieseslesicskslcstesiscocscslescscsvevesesedesesonk tTodkeuicakeaksvskevesieveskaksvleveslcslesicakeskeveseskeskeslcstevedevescakeseslevescseseveseses's 


OPEN(UNIT=1, FILE='LETTER' , STATUS='OLD' ) 
DO 160 I=1, 64 
PATT(I,JMAX) = YC(I) 
CONTINUE 
REWIND 1 
DO 170 I=1, 64 
WRITE(1,175) (PATT(I,J), J=1,COUNT) 
CONTINUE 
FORMAT(1X,1015) 
SUM4 = 0 
BO 1is0 I=1, 64 
SUM4 = SUM4 + W(1I,JMAX)*X(I) 
CONTINUE 
DO 190 I=1, 64 
B(I,JMAX) = (W(I,JMAX)*X(1)) / (0.5 + SUM4) 
WCI,JMAX) = W(I,JMAX)*X(I) 
CONTINUE 
GOTO 400 
OEOSE (1) 
CLOSE (2) 
BLOSE (3) 
CLOSE (4) 
CLOSE (5) 
STOP 
END 


Jesesesveseseslesevesedededescseseacdevevedevedesesedesedesedesdcdedesesesevededcsevesesevededededesevevededevesesesde deve ve veveve 


SUBROUTINE VECMAT( ARR, CMAT) 
DIMENSION ARR( 64), CMAT(8,8) 
K = 0 
DOr 220 J=1, & 

DOez50 1-156 


K=K+1 
CMAT(I,J) = ARR(K) 
CONTINUE 
CONTINUE 
RETURN 
END 


Iivkeavdedketesvakedkekkekakakdkdesdkekekkkkcdededeakdedesakvdakdaksededestedcddekkkkeakdkkkkvedkdeesek kak 


SUBROUTINE MATVEC(CMAT, ARR) 
DIMENSION ARR(64), CMAT(8,8) 
f= 0 
DO 310 J=1, 8 
DO 320 I=1, 8 
K= K +1 
ARR(K) = CMAT(I,J) 
CONTINUE 
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310 CONTINUE 
RETURN 
END 


aedroeievessdeweovocaededevesesesescsvsesesedesesetkseskesesedkocsesessleseskoesedesest desc svseskvacvestcdeseseskesesesdestseskesesesk 


C2 2 


SUBROUTINE CHARMAT( MAT, CMAT) 
DIMENSION MAT( 8,8) ,CMAT(8,8) 
CHARACTER*1 TEMP( 8,8) 
Domes, Tee 
DO 340 J=1,8 
CMAT(1,J) = MATOIM) 
340 CONTINUE 
330 CONTINUE 
DO 650 I=1,8 
DO 660 J = 1,8 
IF(CMAT(I,J).EQ.1) THEN 
TEMP(I,J) = '*' 
ELSE 
TEMP(I,J) =' ' 
ENDIF 
660 CONTINUE 
650 CONTINUE 
DO 370 I= 1, 8 
WRITE(6,167)(TEMP(I,J),J=1,8) 


167 FORMAT(8X,10A1) 
370 CONTINUE 

RETURN 

END 


For the first input pattern to the net, we have used the pattern of the letter “C” given 
below. The elements of the matrice representation take on 0 and | values. To make the 
pattern clearer, We have replaced every element of 0 value bv a white pixel and elements 
of | value by black pixels. A compact representation of this pattern 1s shown to the right 


below : 


SO Oo OQ © G2 © 
ORF RP RBH oO 
eS OOoOood0cdr- 
RK OOdOo0d0drF 
KF OOdO000F 
KF Ooo cdc ddrF 
Ke OOooddo0or-e 
Or OOO OF © 


In a similar manner, the pattern representation of the letter “E” used in the simu- 
lation of the net is shown below. Where the left hand side pattern representation of “E” 


is the actual input to the net. 
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COQ CO Goa @ 02 @ 


a) Co Gee 


mo Om a CO 2 1 


eAOOCOOCOCOOOCO 4 


et GO ee) © Ore 


Pe Oe GO - 


astodtinddetnet et es 


Oo Oo Oo OC ae 


ithe pattern of the letter “F” is represented as : 


OO oOo oo Oo © 


oe O22 OO © 


Coe) ) Se) © © 


oma Octo OO © 


mia mS 2 2 © 


a) = Oi) © © 


atte eA et se 


CaO © Onm@ © © 


mnie meise Corrupted version of the pattern “F” is: 


Oooooeoo 


mOoOqoCco 20 Oa 


i Oo CO OO aS 


OO re © Oo ea-c 


Ore OC =o] 


—O Of OC @oe 


Alalial el Se) el eS! = 


OooOo0O0 00 a2@ 


fmmoisier pattern of the letter “F is: 


OOO CO eGo 2 @ 


OO @2O @ @ © 


aes Ln, <> J TR a 1 > I eT a 


SoS eae eS) SD 


man OTA COCO 


—O Om Oe eo © 


add Ft Od ere 


© © O@ O@ 2 © 
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APPENDIX E. THE PARAMETERS FOR THE MLSE NEURAL 
NETWORK 


From [Ref. 3], the MLSE cost function for the time-varving channel can be written 


as 


M M M 
Ful(an}) = =) 2a + > > a Oy ay (E~1) 
f=] a es 


where s, denotes the value of s,_, at time the 7 observation, z, 1s sampled. Dheve@eum 


cients vary With the time and in general 
an = sv), l = J 
and it follows that 


i’, = so, so, = gene 


4 


thus, the symmetry condition \hich is sufficient for stability no longer holds. 
The MLSE cost function can be reformulated such that the synaptic intercon- 
nections are symmetric. Consider the quadratic term of Equation E-1, 
MoM 
Ddsses (E~2) 


i=l k= 


Let a and f be two integers between 1 and M and assume for the moment that « ¥ Pf. 
Then two of the terms in the summation given by Equation E-2, one fori = a,k = 


and the other fori = £,k = a, are 
a, ea ag & ag ge a, 


respectively. Thus, for indices a and £B the summation given by Equation E-2 contains 


the term 


x ] | x 9 
a, cr ag + ag oe a, = ey ay ag \s ( ae + sf) x) + By ay ag Ge ia sa (E— 3) 
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The two terms on the right side of Equation E-3 are identical. Define the modified co- 


eilicient s’,_, as 


; ae 
S i-k = > (s ius + uly (E — 4) 


clearly, the modified coefficients are symmetric independent of time in the sense that 
Seep} 


When i = k, the modified coefficient becomes 


as desired. Also, using the property given by 


the stationary channel case reduces to 


, 


] 
Sep See ae) = Soe 
Therefore, the MLSE cost function can be written with symmetric s’,’s in a general form 
suitable for either stationary or time-varying channels as 
M OM 
J y(ia,,}) =—_- - > a; <i + > >a So k ar 


i=l f=] 


Where the s’,_,’s are given by Equation E-4. Using the MLSE cost function for the 


time-varying channel, the parameters for the MLSE neural network are given by 


k 
| GT ME Say = (92 a5 Sh) = Wy, a = vit) 


Sik 
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APPENDIX F. PROGRAMMING THE MLSE NEURAL NETWORK 


Using Fortran as programing language, the MLSE neural network, described in 
Chapter V, was implemented with the Mainframe, and used to run some simulations for 
different network parameters and different transmission channel conditions (stationary 


or time-varving channel). 


C Tocserte dovere de devere seve ve seve teste ve ve ve ae Fese seve ve ve ae Ne Vege Nese Te4e Te We NE Tes Te Fe Te Fe Te Fe Fe Fe TE Te Te Fe Fe Fe TIE Te Ne TE TE TEI TO 
@ dered THESIS RESEARCH revere 
C deredere SIMULATION PROGRAM OF THE MLSE NEURAL NETWORK eredere 
G dededegs BY M. H. KHAIDAR tevevede 
C Jevsvevededeve dover ve Te ve seve ds Te 7s Fe FEI Ne Ve Fe7s Fe Ne Tee ETE TOTS TEC FEV C Te Ve TET ETON Ve ETC FON Fe TCT NE TENE TOT FOI FOTW Ve GENE IE 
C 

G Jeasvedededeaededededededevedese seeds Vevews weve Vere sere Te vere Veeder Veer eM VeNe Ve Ne WEN Vee TET Te Ve veye Me Ne Te vee ve ve wee 
C * THIS PROGRAM WAS MADE TO IMPLEMENT AND SIMULATE THE MLSE NEU-* 
C * RAL NETWORK FOR A STATIONARY OR TIME-VARYING CHANNEL. THE PR-* 
C %* OGRAM FIRST WILL ASK THE OPERATOR TO ENTER THE DATA NECESSARY* 
C * TO FULLY DESRIBE THE PROBLEM. THEN, THE PROGRAM IS GOING TO * 
C %* ASK THE OPERATOR IF THE PROGRAM IS TO BE RUN FOR A STATIONARY” 
C %* CHANNEL OR A TIME-VARYING CHANNEL. AFTER THE CHOICE IS MADE * 
C %* THE COMPUTER IS GOING TO DISPLAY THE SIMULATION RESULTS. %* 
C * VARIABLE DECLARATIONS : 4 
C * IR(2500) = DATA SEQUENCE OF 2500 BITS OUTPUT OF THE GGUD * 
C ¥ IMSL SUBROUTINE (= A(2600)) % 
C 7 M = THE NUMBER OF NEURONS IN THE NEURAL NETWORK * 
C % INPUT TO THE PROGRAM, THIS TIME Viises * 
C % MITH, MITER, INDEX, XEND, IWK(17), IER, WK(290), TOL, H=ARE* 
C % THE DESCRIPTION PARAMETERS OF THE PROBLEM TO THE * 
C * DIFFERENTIAL EQUATIONS SOLVER DGEAR. * 
C ¥ L = CHANNEL MEMORY IN UNITS OF T (L = 2 FOR THIS % 
C * SIMULATION), INPUT TO THE PROGRAM. % 
C % N = THE NUMBER OF DATA TRANSMITTED (= NR), INPUT TO * 
C % THE PROGRAM. * 
C % I = THE NUMBER OF DATA BITS SHIFTED INTO THE REGIST-* 
C 7% ERS AT ONCE, INPUT TO THE PROGRAM. * 
C 7 COUNT = THE NUMBER OF DATA BITS THAT DIFFER BETWEEN THE * 
C % TRANSMITTED DATA AND THE MLSE NEURAL NET ESTIMATED* 
C 7 DATA, OUTPUT OF THE PROGRAM. * 
C 7 NUMBER = THE NUMBER OF DATA THAT DIFFER BETWEEN THE MLSE * 
C ¥ NEURAL NET ESTIMATED DATA BITS AND THE DIRECT MLSE* 
C % COST FUNCTION CALCULATED DATA BITS, OUTPUT OF THE * 
C 7 PROGRAM. 
C * REG(17) = DATA BITS IN THE 17 REGISTERS. * 
C * G = GAIN FACTOR OF THE NEURAL AMPLIFIERS. 7 
C * VOUT( 17) = THE DATA BITS OUTPUT OF THE NEURAL AMPLIFIERS. * 
C 7 AOUT( 2500) = ALL THE VOUT(17) WILL BE COLLECTED TO FORM THE* 
C % HOLE ESTIMATED DATA BITS CORRESPONDING TO 2500 DA-* 
C 7 TA BITS TRANSMITTED. * 
C * IN( 2500) = THE 2500 DATA POINTS GENERATED BU GGUD SUBROUTI-* 
C * NE TO FORM THE SAMPLES DELTA(C 2500) USED TO DESGRi. 


108 


C * BE THE TIME-VARYING CHANNEL. (BETWEEN 1 AND 21) * 
C *%*  DELTA( 2500) = TIME-VARYING CHANNEL COEFFICIENTS (RANGE BET-* 
C * 70. PeehOw0.. 1). % 
C * R(2500) = THE 2500 GAUSSIAN NOISE SAMPLES USED TO IMPLEME-* 
C * THE PReseNch OF NOISE IN THE CHANNELS. OUTPUT OF * 
C * THE GGNML IMSL SUBROUTINE. ¥ 
C % UC 17) = THE 17 SOLUTIONS OF THE DIFFERENTIAL EQUATIONS * 
C * SOLVER DGEAR AND INPUTS TO THE NEURAL AMPLIFIERS. * 
C * T = TIME. * 
C * PERIOD = BIT DURATION (INVERSE OF THE DATA RATE) ¥ 
C * MUL = A MULTIPLIER USED TO IMPLEMENT THE TIME-VARYING * 
C * CHANNEL (MUL IS VARYING TOO). ¥ 
C * ViGl?) = EQUALS MUL AT A CERTAIN REGISTER. * 
C * = TNOT = TIME DURATION OF THE INTERSYMBOL INTERFERENCE. * 
C * = NNOT = THE SINGLE SIDED SPECTRAL DENSITY OF THE ADDITI-* 
C * VE WHITE GAUSSIAN NOISE N(T). * 
C * GNC 2600) = GAUSSIAN NOISE SAMPLES GENERATED BY THE GGNML * 
C * SUBROUTINE. * 
C * GNREG(17)= GAUSSIAN NOISE SAMLES INTO THE 17 REGISTERS OF * 
C * THE NEURAL NETWORK. * 
C * Z(17) = THE 17 OBSERVATIONS OF THE STATIONARY CHANNEL * 
C 7% CALCULATED AS DESCRIBED IN THE STUDY. 7 
C poe Y( 17) = THE 17 RECEIVED SAMPLES FOR A STATIONARY CHANNEL* 
C ¥ CALCULATED AS DESCRIBED IN THE STUDY. * 
C *%*  YPRIME( 2500) = EQUIVALENT TO Y(17) BUT THIS TIME WHEN * 
C * CALCULATING FOR THE MLSE COST FUNCTION. * 
C * ZPRIME( 2500) = EQUIVALENT TO Z(17), FOR THE MLSE COST * 
C ¥ FUNCTION. * 
C * MLSECF(2500)= THE 2500 SAMPLES GENERATED BY DIRECT Cae 
C ¥ TION OF THE MLSE COST FUNCTION. 

C % SNR = SIGNAL-TO-NOISE RATIO (INPUT TO THE PROGRAM). ¥ 
C * VPRIME(2500) = SAME AS V(17) BUT NOW IT'S FOR THE MLSE COST* 
C % FUNCTION. * 
C * DR = DATA RATE (INPUT TO THE PROGRAM, MAXIMUM 2400). ¥ 
C * FCN = SUBROUTINE DESCRIBING THE M DIFFERENTIAL EQUATIONS* 
C w FCNJ = EXTRA SUBROUTINE BUT NECESSARY. ¥ 
C ¥ WC17,17) = THE SYNAPTIC CONNECTION MATRIX FOR THE NETWORK * 
C % ACTUALLY IT'S AN M BY M MATRIX. % 
C 7 TAU = TIME CONSTANT OF THE CIRCUIT. % 
C ¥e CURRENT Cl7) = Tae 17 INPUT CURRENIS TO THE 17 NEURONS OF * 
C * THE NEURAL NETWORK. ¥ 
C * VINC17) = THE 17 DATA BITS INTO THE 17 REGISTERS OF THE 

C * NEURAL NETWORK. % 
C deve dedeve deve deve deve doves deve de deve fede fede Bese de Tete Fe ePIC INI ILI IIIS 
C 

C 


: INTEGER IR(2500), X, NR, A(2600), M, METH, MITER, INDEX, ANS 
INTEGER IWK(17), IER, L, N, P, COUNT, SUP, MIN, NUMBER, RESP 
INTEGER MAX, REG(17), G, C, D, Q, VOUT(17), AOUT(2500), IN(2500) 
INTEGER DOWN, UP 

REAL R(2500), U(17), WK(290), T, TOL, H, PERIOD, MUL, V(17) 
REAL TNOT, NNOT, GN(2600), GNREG(17), SUM, FACTOR, DELTA( 2500) 
REAL SUM1, Z(17), F, S, SUM2, YPRIME(2500), Y(17), SUMS, SUM6 
REAL SUM3, ZPRIME(2500), SUM4, MLSECF(2500), SNR, SUM7, SUM8 
REAL SUM9, VPRIME(2500), DR 

DOUBLE PRECISION DSEED 
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Ga ea Ca C2 .C) €)°C) 


3000 
I 


EATERNAL FCN, FCNJ 


COMMON W(17,17), TAU, CURRENT(17), VIN(17) 


Yervedvedededovededededeucvevededeaedevevedese deve de skeve de devedededevevescveve sede vescvevedke devedeveveveakcacucakakeseveavacvcvevess 


PRINT*,' NEURAL NETWORK AS A MLSE RECEIVER’ 


PRINT*, | = 
PRINT*,' 





PRINT*,' PLEASE, ENTER YOUR CHOICE, DO YOU WANT TO :' 


PRENG at 


FORMAT(8X,'(1)',' ',' CONTINUE WITH THE PROGRAM'/ 


SSX, (2) eee CUR 
READ* , RESP 
IF(RESP. EQ. 1) THEN 

GOTO 2 
ELSE 

GOTO 3 
ENDIF 
PRINT*,°) 
PRINT*,° ' 
PRINT*, 'PLEASE, ENTER 
READ* ,M 
PRINT*,' PLEASE, ENTER 
READ* , DR 
PRINT*,'PLEASE, ENTER 
READ* , L 
PRINT*, PLEASE, ENTER 
READ* , TAU 
PRINT*, ' PLEASE, ENTER 
READ* , SNR 
PRINT*, PLEASE, ENTER 
READ* ,N 
PRINT*,' PLEASE, ENTER 


READ* , P 
COUNT = 0 
DOWN = 1 
UP =M 

SUP = M 
MIN = 1 
NUMBER = 0 
MAX = M 


THE 
THE 
SHE 
THE 
THE 
THE 


THE 


NUMBER OF NEURONS (M) :' 

DATA RATE DESIRED (HZ) :' 

CHANNEL MEMORY (L) :' 

TIME CONSTANT (TAU) :' 

SNR CIN DB) :' 

NUMBER OF SYMBOL TRANSMITTED (N) :' 


NO. OF SHIFTED SYMBOLS IN THE VREG. (Gee 


Pee dede Tete Te ete RTE IEE TEE REECE RIEEREERTRERERE RENAE REEREREREREREEREREEREBR ERE 
* GENERATION OF THE N DATA SEQUENCE AND THE N GAUSSIAN NOISE * 


* SAMPLES AND PUTTING THE FIRST M DATA BITS AND M NOISE SAMPLES* 
* INTO THE M REGISTERS OF THE NETWORK TO START THE PROCESSING. * 


e/a’ ae 2a = = =a dl anp'ae’'sa@ ar = «a ona =a = =a - = - i fa a eal =a ale 
Vedelededetedetetedetetetetstetetetetetstededevedehetedvedetedededecedede de cese teae aes ae eae ee ge ee ae ge seas eae 407 ee 


PERIOD = 1 / DR 
TNOT = L * PERIOD 
NNOT 
X = 2 
NR = N 

DSEED = 123457. 0D00 


CALL GGUD(DSEED,X,NR, IR) 


DO 10° 1 =a en 


IFCIRCIJSEG 2) aHeN 


(3 * TNOT) / (4 * (10 ** (SNR / 10.0))) 
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ee al ie a al py eal Gp 


ENDIF 
10 CONTINUE 
DSEED = 123457. ODOO 
CALL GGNML(DSEED ,NR,R) 
DO 20 I = 1, MR 
GN(I) = RCI) 
20 CONTINUE 
DO 30 I =1, M 
REG(I) = ACI) 
GNREG(I) = GN(I) 
30 CONTINUE 
PRINT*,' ' 
PRINT*,' ' 
PRINT*,'YOUR DATA IS NOW ENTERED, THE NETWORK IS READY TO BE' 
PRINT*,'SIMULATED. THIS PROGRAM CAN SIMULATE THE MLSE NEURAL' 
PRINT*, 'NETWORK IN TWO CONDITIONS OF TRANSMISSION CHANNEL : ' 
PRINT*, 


PRINT 5 
5 FORMAT(8X,'(1)',' ','IN A STATIONARY CHANNEL’ / 
&8X,'(2)',' ','IN A TIME-VARYING CHANNEL!’ ) 
PRINT*,° ' 
PRINT*,' PLEASE, ENTER YOUR CHOICE : ' 
READ* , ANS 
ERINT*,  * 
PRINT*,° ' 
PRINT 12 
12 FORMAT(12X,'PROCESSING IN PROGRESS .......... PLEASE WAIT’) 
PRINT*,' ' 
PRINT*,° | 
IF(ANS. EQ. 1) THEN 
GOTO 200 
ELSE 
GOTO 500 
ENDIF 


Seovocvesevesboesleacslevedededesededeskedkeslesleskcscckcckvacacseskcslvckcskeakcakcskvakcakcskeacsescckvckcskeakasskestkecescatsicatcatcckeskcotesesescdesese 


* SIMULATION OF MLSE NEURAL NETWORK IN A STATIONARY CHANNEL * 


SCE GIS Ge Sb oe ae ate ator aoe) stele ter tae bbe abe air ar lic isse aie oie ee oe ioe aoe ie ok kei ke ck Gk keke oi oie ok kc ok ok Gi ok ok ke kk kd ed 
200 DO 40 I =1, M 
HO ™50°K = 1, M 
FACTOR = (I - K) * PERIOD 
SUM = SUM + REG(K) * FCFACTOR,TNOT) 
50 CONTINUE 


40 CONTINUE 


FACTOR = (K - I) * PERIOD 
SUM1 = SUM1 + Y(K) * FCFACTOR,TNOT) 
70 CONTINUE 


If 


Ci C2 €o G> ca 


ZCI) = SUM1 
60 CONTINUE 
DO 90 I=1, M 
CURRENT(I) = 2 * Z(1) 
VINCI) = REG(I) 
DO 100 K = iar 


FACTOR = (1 - Kje*seERIOD 
WCI,K) = -2 * SCFACTOR, TNOT,NNOT) 
100 CONTINUE 
90 CONTINUE 
GOTO 1000 


Sevedesesestcseslcsesesestedesdeskeaeseseseseotevesesestcsesesesesk sesksesevesesesesesesesese celeste sesesese sess se se vesesevese st sexes 


* SIMULATION OF MLSE NEURAL NETWORK IN A TIME-VARYING CHANNEL * 
sesesicvescsevevesesedk dese ve dk sesevesese sede sestseakraedeskcck cece cece seven vevevese de te seaesevesevesiens ve slesieviess ve ve vesies ve ve se 


500 NB = 21 
NR = N 
DSEED = 123457. 0D0 
CALL GGUD( DSEED ,NB,NR, IN) 
DO 310 I = 1, NR 
IF(IN(1I). GE. 1. AND. IN(I). LT. 11) THEN 
DELTA(1) == INCL) | lnoe 
ELSEIF(IN(1I).GE. 11. AND. IN(I). LT. 21) THEN 
DELTA(I) = ( IN(I) - 10 ) / 100.0 
ELSE 
DELTA(I) 


0 
ENDIF 
310 CONTINUE 
K=1 
DO 320 I = DOWN , UP 
V(K) = DELTA(I) + 0.9 
K=K+1 
320 CONTINUE 
DOWN = DOWN + M 
UP = UP + M 
IF(UP. GE.N) UP = N 
DO 330 1=1,M™M 
SUMS = 0 
DO 340 K=1,™ 
FACTOR = (I 
MUL = V(K) 
SUMS = SUMS + REG(K) * FPRIME(FACTOR, TNOT, MUL) 
340 CONTINUE 
Y(I) = SUMS + GNREG(I) 


K) * PERIOD 


330 CONTINUE 
DO 460 I=1,M 
SUM6 = 0 


DO 3608, = 1 ol 
FACTOR = (K - I) * PERIOD 
MUL = V(K) 
SUM6 = SUM6 + Y(K) * FPRIME( FACTOR, TNOT , MUL) 
360 CONTINUE 
ZCI) = SUM6 
460 CONTINUE 
DO 370 I-= 1m 


Wz 


Ca C-Ga Gare) Ca CD CD MlQaQrIaaqg 


Cao C2 C22 Co 


380 
370 
1000 


80 


13 


110 


1a i 


CURRENT(I) = 2 * Z(T) 
VINCI) = REG(I) 
DO 380 K=1,M 
FACTOR = (I - K) * PERIOD 


MUL = V(K) 
W(I,K) = -2 * SPRIME( FACTOR, TNOT ,NNOT, MUL) 
CONTINUE 
CONTINUE 
G = 10000 
T= 0.0 
pO 80 I=1, M 
U(I) = 0.0 
CONTINUE 


TOL = 0.00001 
H = 0.000001 
MITER = 0 

METH = 1 

INDEX = 1 

XEND = 5 * TAU 


Joveskevoocsevevcocdcvevesdcdedevevedcdcsdesdvakedeseslcdesvakevescakeveseocveovscdevevedescveveteucacacvesdkcacdlescdesedescacdeveveve seve 


%* AFTER DOING SOME CALCULATIONS NOW WE ARE READY TO CALL DGEAR * 
% THE DIFFERENTIAL EQUATIONS SOLVER TO SOLVE OUR M EQUATIONS. * 


Jervvevevekvedeveveteletesedededvescsededevetedekdevekvedetcdcvedesekcdedvcdescdededesedevetedededcdevedtevedededodcdevetevede 


CALL DGEAR(M,FCN,FCNJ,T,H,U,XEND,TOL, METH ,MITER, INDEX, IWK,WK, IER) 
IF(IER. GT. 128) THEN 
PRINT 13,IER 
FORMAT(/,' WARNING !!!!!...IER = ' IS) 
GOTO 3 
ENDIF 


aeaktacvevesevessleakcacseakesleveskeaksevedksesesescve se teokeakcdcaksleskeoevescsdeveskcscac sea akokeakvslevevedcakeskeacdevede alee slcs's at ake ak oe 


“THE OUTPUT OF DGEAR, SOLUTIONS TO OUR M DIFFERENTIAL EQUATIONS 
*ARE PASSED THROUGH THE M NEURAL AMPLIFIERS TO GET THE MLSE NEU* 
*RAL NETWORK ESTIMATES OF THE M DATA BITS THAT ARE IN THE M REG* 
*ISTERS. ve 
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DO 110 I1=1, M 
VOUT(I) = -TANH(G * U(I)) 
CONTINUE 
DO1lll1I=1,™ 
IF(REG(I). EQ. 0) VOUT(I) = 0 
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*HERE THE DATA BITS OUTPUTS OF THE NEURAL AMPLIFIERS ARE COLLE-* 
*CTED SO THAT LATER WE ARE GOING TO HAVE THE HOLE 2500 ESTIMATE* 
*S OF THE 2500 DATA BITS TRANSMITTED. * 


POUT TEETER TER TE TEES TOL TETOTETS TE TS CTT TEREST TTC TE CTE TCT TS CTE TENT TOTTE TELE EEE TELTE RRR EERIE 


IF( MAX. EQ.M) THEN 

K= 1 
Py ol 
VOUT( J) 
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120 


165 


166 
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CONTINUE 
ELSEIF(MAX. NE.N) THEN 
DO 116 J = M-P , M-1 
AOUT(K) = VOUT(J) 
fil 
CONTINUE 
ELSEIF(MAX. EQ.N) THEN 
MPRIME = M - P 
DO 118 I=M,1, -1 
IF(REG(I).EQ.0) THEN 
M=M-1 
ENDIF 
CONTINUE 
IF(M. GT. MPRIME) THEN 
DO 117 J = MPRIME , M 
AOUT(K) = VOUT(J) 
K=kil 
CONTINUE 
ENDIF 
ENDIF 


Jesicstevescdesesedecdeskvcdkccltesicvesescotsestestelestedesedsdesvedesesstccleseskccestedcceslssesdesesesese stele testes desc tedestcdesesevedeses’s 


*HERE, THE ESTIMATES FROM NEURONS L+1 THROUGH M-1 ARE TAKEN AS * 
*VALID. THEN A COMPARISON BETWEEN THESE ESTIMATES AND THEIR COR* 
*RESPONDING IN THE REGISTERS I.E. THE INPUT DATA BITS TS Apis 
*AND THE RESULT IS RECORDED FOR LATER USE. ¥ 


Sevevesievesesesesessvesesssecleslesevededescsevedesevescveveslevededesvvevovedtsesevededesededesedovededesese deve devesede sesso deve 


C ieee 
= 
TE CMAX. EQ2MO Raia 
DO-1205 1 = sire t 
IFC VINCI). NESYOUT( ip), THEN 
COUNT = COUNT + 1 


ENDIF 
CONTINUE 
ELSEIF( MAX. EQ.N) THEN 
DO@iGael = (Niaige=1 
PGR EG. 1) eB OS Omen ain 


M=M-1 
ENDIF 
CONTINUE 
DO 166 I=C, M 
IF(VIN(I).NE. VOUT(I)) THEN 
COUNT = COUNT + 1 
ENDIF 
CONTINUE 
ELSE 
DO 130 I=C, D 
IF(VIN(1I).NE. VOUT(I)) THEN 
COUNT = COUNT + 1 
ENDIF 
CONTINUE 
ENDIF 
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“Tee eee DATA BITS IN THE REGISTERS FROM 1 TO P ARE NULLED * 
*THEN THE CONTENT OF THE SHIFT REGISTERS ARE SHIFTED TO THE RIG* 
*HT TILL THE CONTENT OF REGISTER 1 IS NONZERO. THEN, WE ARE FEE* 
SOeicetm: Foie lrennoletins BY THE NEAT P DATA BITS OF THE TRAN* 


*SMITTED SEQUENCE FOR ANOTHER PROCESSING CYCLE. ¥ 
Selevevesedededesedededeseskesevedescslesedesedeslevesedeseseskesevestecesededesledevesesedecleslestcscskcslvaecesksesesestecesrseseslevevesc 


DO 140 I=1, P 


CONTINUE 
Q= 1 
IF(Q. LE.M) THEN 
IF((P+Q). LE.M) THEN 
REG(Q) = REG(P+Q) 
GNREG(Q) = GNREG(P+Q) 
ELSE 
REG(Q) = 0 
GNREG(Q) = 
ENDIF 
Oh [Ojecoal 
GOTO 150 
ENDIF 
IF(MAX. EQ.N) THEN 
PRINT*,'THE NUMBER OF ERROR DATA BETWEEN THE TRANSMITTED' 
PRINT*, ' BINARY SEQUENCE AND THE MLSE NEURAL NET OUTPUT' 
PRINT*,'DATA IS :' 
PRINT*,¢ ' 
PRINT 270, COUNT 
EORMAT( 2X, COUNT =',2X,15) 
GOTO 2000 
ENDIF 
mOe16co I = 1, M 
IF(REG(1).EQ.0) THEN 
MAX = MAX + 1 
IF(MAX. GT.N) THEN 
AC(MAX) = 0 
GN(MAX) = 0 
ENDIF 
REG(I) = A(MAX) 
GNREG(I) = GN(MAX) 
ENDIF 
CONTINUE 
IF(MAX.GT.N) THEN 
io N 
ENDIF 
IF(ANS. EQ. 1) THEN 
GOTO 200 
ELSE 
GOTO 500 
ENDIF 
IF( ANS. NE. 1) THEN 
GOTO 300 
ENDIF 
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700 


190 


180 


240 


230 


260 
250 


300 


470 


410 


590 


430 


420 


*AFTER, THE MLSE NEURAL NETWORK HAS ESTIMATED THE N DATA BITS * 
*TRANSMITTED, WE ARE NOW GOING TO START CALCULATING THE DIRECT * 
*MLSE COST FUNCTION ESTIMATES OF THE N TRANSMITTED DATA BITS * 
*THEN, WE ARE GOING TO MAKE A COMPARISON BETWEEN THE TWO ESTIMA* 
*TES AND RECORD THE RESULT. ¥ 
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DO 180 I = MIN, SUP 
SUM2 = 0 
DO 190 K = MIN, SUP 
FACTOR = (I - K) * PERIOD 
SUM2 = SUM2 + ACK) * F(FACTOR,TNOT) 
CONTINUE 
YPRIME(I) = SUM2 + GN(I) 
CONTINUE 
DO 230 I = MIN, SUP 
SUM3 = 0 
DO 240 K = MIN, SUP 
FACTOR = (K - I) * PERIOD 
SUM3 = SUM3 + YPRIME(K) * F(FACTOR,TNOT) 
CONTINUE 
ZPRIME(1) = SUM3 
CONTINUE 
DO 250 I = MIN, SUP 
SUM4 = 0 
DO 260 K= MIN, SUP 
FACTOR = (I - K) * PERIOD 
SUM4 = SUM4 + A(I) * S(FACTOR,TNOT,NNOT) * ACK) 
CONTINUE 
MLSECF(I) = 2 * ACI) * ZPRIME(I) - SUM4 
CONTINUE 
GOTO 350 
a 
DO 470 I = MIN , SUP 
VPRIME(I) = V(J) 


Jd ee 
CONTINUE 
DO 390 I> = MING our 
SUM7 = 0 
DO 410 K = MIN , SUP 
FACTOR = (I - K) * PERIOD 
MUL = VPRIME(K) 
SUM7 = SUM7 + ACK) * FPRIME(C FACTOR, TNOT , MUL) 
CONTINUE 
YPRIMEC]) = SUti7va+ chap 
CONTINUE 
DO 420 I = MIN , SUP 
SUM8 = 0 
DO 430 K = MIN , SUP 
FACTOR = (CK - 1)5* BPERDOD 
MUL = VPRIME(K) 
SUM8 = SUM8 + YPRIME(K) * FPRIME( FACTOR, TNOT, MUL) 
CONTINUE 
ZPRIME(I) = SUM8 
CONTINUE 


DO 440 I = MIN , SUP 
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SUM9 = 0 
DO 450 K = MIN , SUP 
FACTOR = (I - K) * PERIOD 
MUL = VPRIME(K) 
SUM9 = SUM9 + A(I)*SPRIME( FACTOR, TNOT,NNOT,MUL)*A(K) 
CONTINUE 
MLSECF(I) = 2 * ACI) * ZPRIME(I) - SUM9 
CONTINUE 
IF(SUP.EQ.N) GOTO 400 
MIN = MIN + M 
SUP = SUP + M 
IF(SUP. GT.N) THEN 
SUP =N 
ENDIF 
IF(ANS. EQ. 1) THEN 
GOTO 700 
ELSE 
GOTO 300 
ENDIF 
DO 280 I=I1, N 
IF(AOUT(I).NE.MLSECF(I)) THEN 
NUMBER = NUMBER + 1 
ENDIF 
CONTINUE 
PRINT*,'’THE NUMBER OF ERROR DATA BETWEEN THE TRANSMITTED ' 
PRINT*,' BINARY SEQUENCE OF DATA AND DATA DIRECTLY GENERATED ' 
PRINT*,'BY THE MLSE COST FUNCTION : ' 
PRINT*,° | 
PRINT 290,NUMBER 
FORMAT( 2X, ‘NUMBER =' ,4X,15) 
GOTO 3000 
STOP 
END 
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“HERE IS THE SUBROUTINE CORRESONDING TO THE TRANSMISSION CHANNE* 
*I, IMPULSE RESPONSE WHICH IS MODELED BY A FINITE RESPONSE SQUAR* 
“ED COSINE FUNCTION. THIS FUNCTION IS IMPLEMENTING THE STATIONA* 
“RY CHANNEL. x 


UJ 
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FUNCTION F( FACTOR, TNOT) 
REAL FACTOR 
REAL F, TNOT, PI, SUP 
PI = 3.1415927 
SUP = TNOT / 2.0 
IF(ABS( FACTOR). LE. SUP) THEN 
F = (COS(PI *FACTOR / TNOT)) ** 2 


ELSE 
F = 0 
ENDIF 
RETURN 
END 
Mastered deve sesesevede Te ve teste Keke de vevede Ke ve dede dese Te ke Tete Kee I III RRR RR RII 
*HERE IS THE COMBINED RESPONSE OF THE CHANNEL AND MATCHED w 
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*FILTER. THIS FUNCTION IS IMPLEMENTING THE STATIONARY CHANNEL. * 
akaevedeseslssiestcaesleslssclcaveclcaatccleslcstcevesktatbotscsestvleatacaecesklcueveveskvobsievesdeslesosisescstacdesesksesesesesvakvestscdese 


FUNCTION S( FACTOR, TNOT,NNOT) 
REAL FACTOR, S, TNOT, NNOT, PI 
PI = 3. 1415927 
IF(ABS(FACTOR). LE. TNOT) THEN 
= (1 / (2 * NNOT)) * ((TNOT - ABS(FACTOR)) 


* (1 + 0.5 * COS(2 * PI * FACTOR / TNOT)) 
+ (3 * TNOT / 4.0) * SIN (2 * PI * ABS(FACTOR) * TNOT)) 
ELSE 
S=0 
ENDIF 
RETURN 
END 
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*HERE ARE THE SET OF DIFFERENTIAL EQUATIONS DESCRIBING THE DYNA* 


*MICS OF THE NEURAL NETWORK. ve 
bie Loe hi) ae wir) ie) eee Le hoe hae Lode irk eh Sieh ed eh Sera eka oe oe hohe eek em Lor) ok hoi oe) whe oe hard we lorioe lor hori ihorisrlorhorirelorisrlorlorioriorlorharlorlsriorisrisrisrls 


SUBROUTINE FUNC M, TP yU{UPRIME) 
INTEGER M 
REAL U(M), UPRIME(M), T, SOM 
COMMON WC17,17), TAU, CURRENT( 17), VIN(17) 
pO 210 I=1,M 

SOM = 0 

DO 2205k = 13° 

SOM = SOM + WCI,K) * VINCK) 

CONTINUE 

UPRIMECI) = SOM - (UCI) / TAU) + CURRENTCI) 
CONTINUE 
RETURN 
END 
tededededededcdekdedcdeteksescacdedesdedededeteakdesetevese Teva a seve de ve deseuesc ve ae ak a0 20 26 ae 0 20 oC C Ie ae ae ae oh oe oc ob ee ae 
SUBROUTINE’ FONJICM, TU, PD) 
INTEGER M 
REAL UC) [@ebG)- & 
RETURN 
END 
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* SAME AS FOR THE FUNCTION F ONLY THIS TIME IS FOR THE TIME- VARY 


*“ING CHANNEL. 
sevcdedcdcdededvacdesevedesescdcakcavdtdedcscseskcsesescskvakasescsdevedkscsdesesdcakcseskdedescsksicakc sede se deatkscaesedc akeaeveve seve ve 


FUNCTION FPRIME( FACTOR, TNOT, MUL) 
REAL FACTOR 
REAL FPRIME, TNOT, PI, SUP, MUL 
PI = 3.1415927 
SUP = TNOT / 2.0 
IF(ABS( FACTOR). LE. SUP) THEN 
FPRIME = MUL * ((COS(PI *FACTOR / TNOT)) *#* 2) 
ELSE 
FPRIME = 0 
ENDIF 
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RETURN 
END 
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*SAME AS BEFORE (FOR THE FUNCTION S) ONLY THIS TIME IT IS FOR * 


*THE TIME-VARYING CHANNEL. se 
seasdoucdedestcacvdedescdedkesesesesededoaeseseskacskeskdkatscdestcakvskeslesestsescacskvskcdcacvesvoesk kc oksesesksesesestsksesescvetecesc 


FUNCTION SPRIME( FACTOR ,TNOT,NNOT , MUL) 
REAL FACTOR, SPRIME, TNOT, NNOT, PI, MUL 
PI = 3.1415927 
IF(ABS(FACTOR). LE. TNOT) THEN 
SPRIME = ((MUL ** 2) / (2 * NNOT)) * ((TNOT - ABS(FACTOR)) 


& Plena COo(2 * Pi « FACTOR / TNOT)) 
& + (3 * TNOT / 4.0) * SIN (2 * PI * ABS(FACTOR) * TNOT)) 
ELSE 
SPRIME = 0 
ENDIF 
RETURN 
END 
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